Abstract

Pretrained Vision Transformers (ViTs) such as DINOv2 and MAE provide generic image features that can be applied to a variety of downstream tasks such as retrieval, classification, and segmentation. However, such representations tend to focus on the most salient visual cues in the image, with no way to direct them toward less prominent concepts of interest. In contrast, Multimodal LLMs can be guided with textual prompts, but the resulting representations tend to be language-centric and lose their effectiveness for generic visual tasks. To address this, we introduce Steerable Visual Representations, a new class of visual representations, whose global and local features can be steered with natural language. While most vision-language models (e.g., CLIP) fuse text with visual features after encoding (late fusion), we inject text directly into the layers of the visual encoder (early fusion) via lightweight cross-attention. We introduce benchmarks for measuring representational steerability, and demonstrate that our steerable visual features can focus on any desired objects in an image while preserving the underlying representation quality. Our method also matches or outperforms dedicated approaches on anomaly detection and personalized object discrimination, exhibiting zero-shot generalization to out-of-distribution tasks.

Abstract

Audio Descriptions (ADs) narrate visual content for Blind and Low Vision (BLV) audiences during gaps in audiovisual media. There is growing momentum around ADs in movies and TV shows, and with mandates from India's Central Board of Film Certification (CBFC), there is a need to expand ADs beyond English. Yet, there is no work that generates ADs for any Indian language. To address this gap, we present the first systematic study of ADs in Hindi, contributing to aspects such as data, generation, and evaluation. We introduce Andha-Dhun, the first dataset of human-authored Hindi ADs collected from 8 full-length movies. We explore two approaches for generating ADs in Hindi: (i) directly from English dense video descriptions, and (ii) translating English ADs into Hindi. We evaluate these approaches using perplexity and LLM-as-a-judge metrics to assess fluency and quality respectively. We also analyze movies that have both English and Hindi human-authored ADs and find that naïve translation introduces artifacts and narrows diversity compared to original Hindi ADs. Direct machine translation fails to adapt cultural references, while human-translated ADs do better but still fall short. Our findings emphasize that the purpose of Hindi ADs is accessibility for Indian BLV audiences, and that this requires adapting content for the audience more than strict fidelity to the source.

Abstract

Subtitles are essential for accessibility and global dissemination of video content, yet conventional formats often omit critical cues such as speaker identities. Character-Aware Subtitling (CAS) aims to address this gap, however, existing approaches are limited by the absence of public checkpoints, weak pipeline coordination leading to detrimental performance, and insufficient failure analysis. This work presents a systematic study of subtitling pipelines, demonstrating that improved transcript-speech alignment substantially enhances downstream performance, achieving state-of-the-art results on the LLR dataset by 2.77% DER-point zero-shot, and by 4.78% DER-point after VAD finetuning. In addition, we propose design refinements, adopt open, out-of-the-box model alternatives, and introduce a scalable method for curating diverse character priors, thereby improving the robustness and replicability of CAS.

Abstract

We study the connection between audio observations and the underlying physics of a mundane yet intriguing everyday activity: pouring liquids. Given only the sound of liquid pouring into a container, our objective is to automatically infer physical properties such as the liquid level, the shape and size of the container, the pouring rate and the time to fill. To this end, we: (i) show in theory that these properties can be determined from the fundamental frequency (pitch); (ii) train a pitch detection model with supervision from simulated data and visual data with a physics-inspired objective; (iii) introduce a new large dataset of real pouring videos for a systematic study; (iv) show that the trained model can indeed infer these physical properties for real data; and finally, (v) we demonstrate strong generalization to various container shapes, other datasets, and in-the-wild YouTube videos. Our work presents a keen understanding of a narrow yet rich problem at the intersection of acoustics, physics, and learning. It opens up applications to enhance multisensory perception in robotic pouring.

Abstract

Adapting CLIP for videos has gained popularity due to its semantic and rich representation. While CLIP is a good starting point, it typically undergoes post-pretraining (contrastive finetuning) on large video narration or caption datasets (e.g. HowTo100M, WebVid2.5M). However, such narrations or captions often lack comprehensive information needed to represent a video holistically. As the learning signal from text is sparse, the visual learning is inefficient and adaptation requires millions of samples to post-pretrain. In this work, we ask: is it possible to efficiently adapt CLIP for general and holistic video understanding? We use videos labeled with structured and dense Semantic Role Labels (SRLs) that capture actions, people or objects, their attributes, adverbs (manner), and location in a structured format representing the entire video in a holistic way. We generate rule-based captions from SRLs and demonstrate that simple contrastive finetuning on a mere 23k video-caption pairs is adequate to learn powerful, transferable representations applicable across a diverse range of video understanding tasks that require varying levels of perceptual granularity. Our adapted CLIP model, SRL-CLIP, exhibits comparable or superior performance on zero-shot text-to-video retrieval compared to state-of-the-art models that possess 4-8x more parameters and are post-pretrained on up to 6000x more data. SRL-CLIP surpasses CLIP on multiple video benchmarks, underscoring the efficient learning and improved representations.

Abstract

Video Situation Recognition (VidSitu) addresses the challenging problem of "who did what to whom, with what, how, and where" in a video. It tests thorough video understanding by requiring identification of salient actions and associated short descriptions for event roles across multiple events. Grounding with VidSitu requires spatio-temporal localization of key entities across shots and varied appearances.
We posit that coherent video understanding requires consistent identification of entities that play different roles. We propose Multimodal Entity Coreference (MEC) to unite entity descriptions in text with grounding across the video. Towards this, we introduce CineMEC, a multi-stage approach that unites event role mention groups with visual clusters of entities, without explicit grounding supervision during training. Our approach is designed to exploit the synergy between visual grounding and captioning, where improving one influences the other and vice versa. For evaluation, we extend the VidSitu dataset with grounding annotations. While previous work focuses primarily on descriptions, CineMEC improves consistency across both: captioning (+2.5% CIDEr, +7% LEA) and visual grounding (+18% HOTA).

Abstract

Text as signs, labels, or instructions is a critical element of real-world scenes as they can convey important contextual information. 3D representations such as 3D Gaussian Splatting (3DGS) struggle to preserve fine-grained text details, while achieving high visual fidelity. Small errors in textual element reconstruction can lead to significant semantic loss. We propose STRinGS, a text-aware, selective refinement framework to address this issue for 3DGS reconstruction. Our method treats text and non-text regions separately, refining text regions first and merging them with non-text regions later for full-scene optimization. STRinGS produces sharp, readable text even in challenging configurations. We introduce a text readability measure OCR Character Error Rate (CER) to evaluate the efficacy on text regions. STRinGS results in a 63.6% relative improvement over 3DGS at just 7K iterations. We also introduce a curated dataset STRinGS-360 with diverse text scenarios to evaluate text readability in 3D reconstruction. Our method and dataset together push the boundaries of 3D scene understanding in text-rich environments, paving the way for more robust text-aware reconstruction methods.

Abstract

Background
Despite showing remarkable performance on tasks across modalities, deep neural networks remain opaque, with limited insight on how they internally organize information. A learned hierarchy is suggested in vision (Zhou et al., 2015) and language (Tenney et al., 2019) models in deep learning.
Bregman (1990) describes the process of auditory scene analysis (ASA), where the auditory system abstracts information based on perceptual properties such as pitch and timbre, among others, in order to construct auditory streams that are then integrated to form higher order percepts like musical genre. This implies that the representation of audio goes through a series of hierarchical abstractions in the process. While hierarchical representations are indicated in human auditory processing (Kell et al., 2018), the same has not been thoroughly demonstrated in deep learning models for audio.

Aims
We study how convolutional neural networks (CNNs) represent the hierarchy of audio tasks, hypothesizing that representations at earlier layers of a CNN perform better at lower level tasks, and those at later layers perform better at higher level tasks.

Methods
On the basis of Bregman's ASA model, we choose tasks that are arranged in a hierarchy from the domains of speech and music. We select three hierarchical musical tasks - note identification, instrument classification, and genre classification, representative of low-, mid-, and high-level tasks. We use class-balanced subsets of NSynth (Engel et al., 2017) with 1800 instances across 12 classes, Medley-solos-DB (Lostanlen et al., 2019) with 965 instances across 7 classes , and GTZAN (Tzanetakis et al., 2001) with 1000 instances across 10 classes, for each of the respective tasks.
We inspect three models - VGGish (Hershey et al., 2017), CLAP (Elizalde et al., 2023), and MobileNetV3 (Schmid et al., 2023). We use k-Nearest Neighbour classifiers with class-balanced five-fold cross-validation to assess the model's accuracy on each task with intermediate representations extracted at six equally spaced convolutional blocks and the first fully-connected layer. We repeat this for three hierarchically related speech tasks - consonant classification, keyword recognition, and speaker count estimation, using similarly class-balanced subsets of PCVC (Malekzadeh et al., 2020) with 1794 instances across 23 classes, Speech Commands (Warden, 2018) with 1750 instances across 35 classes, LibriCount (Stöter et al., 2018) with 1100 instances across 11 classes respectively.

Results
Our results show that the CNN's early layers learned low-level tasks and later layers learned high-level tasks even without explicit training. On CLAP, for note identification, the first layer's accuracy (47%) is greater than later layers (<30%). For instrument classification, we see peak accuracy at the last layer (97%), but a remarkable change occurs at the fourth layer (71%) compared to the third layer (55%). Genre classification accuracy at the last layer (75%) is better than earlier layers (<66%). These trends hold across all models. We see similar trends for the speech tasks.

Discussion
The results support our hypothesis, providing strong evidence that CNNs implicitly learn a hierarchy in sound. This mirrors the human brain's hierarchical encoding and deserves further investigation if other deep learning architectures and training methodologies also encode similar hierarchies. We also see how low-level tasks are implicitly learnt.

Abstract

Recent advances in text-to-image models have enabled a new era of creative and controllable image generation. However, generating compositional scenes with multiple subjects and attributes remains a significant challenge. To enhance user control over subject placement, several layout-guided methods have been proposed. However, these methods face numerous challenges, particularly in compositional scenes. Unintended subjects often appear outside the layouts, generated images can be out-of-distribution and contain unnatural artifacts, or attributes bleed across subjects, leading to incorrect visual outputs. In this work, we propose MALeR, a method that addresses each of these challenges. Given a text prompt and corresponding layouts, our method prevents subjects from appearing outside the given layouts while being in-distribution. Additionally, we propose a masked, attribute-aware binding mechanism that prevents attribute leakage, enabling accurate rendering of subjects with multiple attributes, even in complex compositional scenes. Qualitative and quantitative evaluation demonstrates that our method achieves superior performance in compositional accuracy, generation consistency, and attribute binding compared to previous work. MALeR is particularly adept at generating images of scenes with multiple subjects and multiple attributes per subject.

Abstract

Audio descriptions (ADs) narrate important visual details in movies, enabling Blind and Low Vision (BLV) users to understand narratives and appreciate visual details. Existing works in automatic AD generation mostly focus on few-second trimmed clips, and evaluate them by comparing against a single ground-truth reference AD. However, writing ADs is inherently subjective. Through alignment and analysis of two independent AD tracks for the same movies, we quantify the subjectivity in when and whether to describe, and what and how to highlight. Thus, we show that working with trimmed clips is inadequate. We propose ADQA, a QA benchmark that evaluates ADs at the level of few-minute long, coherent video segments, testing whether they would help BLV users understand the story and appreciate visual details. ADQA features visual appreciation (VA) questions about visual facts and narrative understanding (NU) questions based on the plot. Through ADQA, we show that current AD generation methods lag far behind human-authored ADs. We conclude with several recommendations for future work and introduce a public leaderboard for benchmarking.