Abstract

Pedestrian intention and trajectory prediction are critical for the safe deployment of autonomous driving systems, directly influencing navigation decisions in complex traffic environments. Recent advances in large vision-language models offer a powerful new paradigm for these tasks by combining high-capacity visual understanding with flexible natural language reasoning. In this work, we introduce PedestrianQA, a large-scale video-based dataset that formulates pedestrian intention and trajectory prediction as question-answering tasks augmented with structured rationales. PedestrianQA expresses richly annotated pedestrian sequences, in natural language, enabling VLMs to learn from visual dynamics, contextual cues, and interactions among traffic agents while generating concise explanations of their predictions without needing specialized architectures tailored for each task. Empirical evaluations across PIE, JAAD, TITAN, and IDD-PeD show that finetuning state-of-the-art VLMs on PedestrianQA significantly improves intention classification, trajectory forecasting accuracy, and the quality of explanatory rationales, demonstrating the strong potential of VLMs as a unified and explainable framework for safety-critical pedestrian behavior modeling.

Abstract

Visual Forced Alignment (VFA) aims to temporally localize linguistic units within silent talking-face videos, enabling alignment when audio is noisy or unavailable. We propose V-Align, a structured framework that formulates VFA as an optimal monotonic path traversal problem over a frame-phoneme compatibility lattice. Given the lip-motion embeddings and transcript-derived phoneme representations, V-Align learns a soft traversal posterior and recovers discrete boundaries via dynamic programming. Training follows a two-stage strategy: Stage 1 encourages optimal path learning without explicit boundary supervision and already achieves competitive performance relative to fully supervised VFA baselines, while Stage 2 refines boundary precision using additional word-level supervision. Experiments on LRS2 and LRS3 demonstrate state-of-the-art MAE and frame accuracy. Code and demo are available at: https:
//valign-interspeech.github.io/.

Abstract

Lip-to-Speech synthesis aims to generate intelligible, natural-sounding, and temporally synchronized speech from silent visual recordings of a speaker. Although recent end-to-end models have shown promising results, they require large amounts of training data and often lack out-of-dataset generalization.
In this work, we introduce LipAdapter, a generic and efficient modular framework that adapts an existing frozen pre-trained Text-to-Speech model into a lip-synchronized speech generator. LipAdapter bridges the modularity gap between frozen Visual
Speech Recognition and Text-to-Speech models by aligning phoneme representation and lip embeddings via a novel text-to-video alignment module. Our model achieves state-of-the-art performance across multiple benchmarks in English while using 15× less training data than previous methods. We also demon-
strate the model's zero-shot capabilities in multilingual lip-to-speech synthesis in French, German, Spanish and Portuguese. Code and demo are available at: https://lipadapter.github.io/.

Abstract

Abstract-- Two-wheelers account for a disproportionately
high share of road fatalities in the Global South. Research
on two-wheeler rider behavior, however, lags far behind fourwheelers, where multimodal datasets have driven major advances in Advanced Driver Assistance Systems (ADAS). To
address this gap, we present the MOtorized TwO-wheeler Rider
(MOTOR) dataset, the first large-scale, multi-view, multimodal
resource dedicated to two-wheelers in dense, unstructured
traffic. MOTOR comprises 1,629 sequences (25+ hours of video
data) collected from 16 riders and integrates synchronized
front, rear, and helmet videos, rider eye-gaze from wearable
trackers, on-road audio, and telemetry (GPS, accelerometer,
gyroscope). Rich annotations capture traffic context, rider state,
12 riding maneuvers spanning conventional and unconventional
behaviors, and legality labels (Legal, Illegal, Unspecified). We
benchmark rider behavior recognition and maneuver legality classification using state-of-the-art video action recognition backbones (CNN and Transformer-based), extended with
multimodal fusion, and find that combining RGB, gaze, and
telemetry consistently yields the best performance. MOTOR
thus provides a unique foundation for advancing safety-critical
understanding of two-wheeler riding. It offers the research
community a benchmark to develop and evaluate models for
behavior analysis, legality-aware prediction, and intelligent
transportation systems. Dataset and code is available at https:
//varuniiith.github.io/MOTOR-Dataset/

Abstract

The communication of scientific knowledge has become increasingly multimodal, spanning text, visuals, and speech through materials such as research papers, slides, and recorded presentations. These different representations collectively convey a study's reasoning, results, and insights, offering complementary perspectives that enrich understanding. However, despite their shared purpose, such materials are rarely connected in a structured way. The absence of explicit links across formats makes it difficult to trace how concepts, visuals, and explanations correspond, limiting unified exploration and analysis of research content.
To address this gap, we introduce the Multimodal Conference Dataset (MCD), the first benchmark that integrates research papers, presentation videos, explanatory videos, and slides from the same works. We evaluate a range of embedding-based and vision-language models to assess their ability to discover fine-grained cross-format correspondences, establishing the first systematic benchmark for this task. Our results show that vision-language models are robust but struggle with fine-grained alignment, while embedding-based models capture text-visual correspondences well but equations and symbolic content form distinct clusters in the embedding space. These findings highlight both the strengths and limitations of current approaches and point to key directions for future research in multimodal scientific understanding. To ensure reproducibility, we release the resources for MCD at https://github.com/meghamariamkm2002/MCD

Abstract

This paper addresses a key challenge in Handwritten Text Recognition (HTR): the dependence on large volumes of labeled data. To overcome this, we propose a self-supervised learning (SSL) framework, LoGo-HTR, that minimizes labeling requirements while achieving strong recognition performance. We introduce a large-scale dataset, SSL-HWD of 10 million word-level handwritten images from diverse scanned documents, partitioned into a small labeled subset and a much larger unlabeled subset. LoGo-HTR combines a local contrastive loss for spatial consistency and a global decorrelation loss to enhance feature diversity. This dual objective enables robust, invariant, and spatially discriminative feature learning. After self-supervised pretraining, we fine-tune a transformer-based decoder using limited labeled data. Extensive experiments on standard HTR benchmarks, which include multilingual and historical data, demonstrate that, after SSL pretraining on our unlabeled dataset, our method consistently outperforms state-of-the-art approaches, even when fine-tuned using only 80% and 20% of the available labeled training data from the respective benchmarks. Ablation studies highlight the effectiveness of our dual loss design and demonstrate the potential of scalable, label-efficient handwritten text recognition. The SSL-HWD dataset and LoGo-HTR model with code are publicly available at https://logo-ssl.github.io/.

Abstract

Generative AI models, particularly Text-to-Video (T2V) systems, offer a promising avenue for transforming science education by automating the creation of engaging and intuitive visual explanations. In this work, we take a first step toward evaluating their potential in physics education by introducing a dedicated benchmark for explanatory video generation. The benchmark is designed to assess how well T2V models can convey core physics concepts through visual illustrations. Each physics concept in our benchmark is decomposed into granular teaching points, with each point accompanied by a carefully crafted prompt intended for visual explanation of the teaching point. T2V models are evaluated on their ability to generate accurate videos in response to these prompts. Our aim is to systematically explore the feasibility of using T2V models to generate high-quality, curriculum-aligned educational content--paving the way toward scalable, accessible, and personalized learning experiences powered by AI. Our evaluation reveals that current models produce visually coherent videos with smooth motion and minimal flickering, yet their conceptual accuracy is less reliable. Performance in areas such as mechanics, fluids, and optics is encouraging, but models struggle with electromagnetism and thermodynamics, where abstract interactions are harder to depict. These findings underscore the gap between visual quality and conceptual correctness in educational video generation. We hope this benchmark helps the community close that gap and move toward T2V systems that can deliver accurate, curriculum-aligned physics content at scale. The benchmark and accompanying codebase are publicly available https://github.com/meghamariamkm/PhyEduVideo.

Abstract

Sign Language Production (SLP) systems can significantly improve accessibility for Deaf communities by translating spoken or written language into sign language videos. For millions of Indian Sign Language (ISL) users, such systems could bridge persistent communication gaps in education, healthcare, and public services. However, SLP in ISL is hindered by the scarcity of annotated datasets and the expressive complexity of the language. Existing ISL datasets, such as iSign cite{joshi2024isign} and ISL-CSLRT cite{elakkiya2021islcsltr}, are designed for recognition and lack gloss-levelfootnote{Gloss is word representation of sign.} annotations, forcing full-sentence modeling that struggles to generalize to new constructions.

We propose an interpolation-based approach that stitches together individual gloss-level signs using modern text-to-motion frameworks like FramePack cite{zhang2025framepack}. This produces contextually accurate and grammatically consistent sign sequences while retaining fine-grained articulation. To support this pipeline, we curate a 1,000-sentence ISL dataset, translated by a native interpreter and annotated with precise gloss boundaries. Results show that with high-quality gloss-level supervision, interpolation-based synthesis offers a practical, scalable path for inclusive SLP in low-resource sign languages like ISL. The project page can be found here url{https://cvit.iiit.ac.in/research/projects/cvit-projects/towards_ssp}.

Abstract

Automated table structure recognition from input images faces significant visual and linguistic challenges, including layout variances, table densities, and the presence of empty or multi-line cells. This problem is crucial due to its extensive applications in business and scientific domains. For this purpose, we present , our end-to-end trainable model using a two-step process for table parsing: Top Down, decomposing the table image into cells, and Bottom Up, associating cells to define the complete structure by assigning row and column spans. This method leverages constraints inspired by human cognition, such as cell alignment, continuity, and non-overlapping characteristics, optimizing the objective function to preserve these attributes. We introduce a vision transformer backbone augmented with local attention tailored for high-resolution input images. We also enhance inference speed for row and column adjacency prediction by replacing sampling-based graph neural networks with a single head self attention layer. Furthermore, we propose an effective split-and-merge strategy to handle highly dense tables without altering the network architecture. Additionally, we present problem-specific training strategies that improve performance across diverse visual characteristics and support cross-domain adaptations. Finally, through our robust protocol for comprehensive dataset analysis and result interpretation, we offer clear directions for future work in automated table structure recognition, highlighting its potential for substantial advancements in practical applications.

Abstract

Financial statements are critical for corporate analysis, fraud detection, and regulatory compliance, yet their complex tabular layouts, multi-year data structures, and hierarchical relationships pose significant challenges for automated reasoning. Existing Visual Question Answering (VQA) models struggle to deliver audit-ready responses due to their inability to explicitly localize evidence and reason over structured financial data. To bridge this gap, we introduce EviFiVQA, a large-scale benchmark dataset for evidence-based financial VQA, built on real-world financial statements. Leveraging FinTabNet, our dataset contains over 1.5 million question-answer pairs spanning five reasoning categories, each meticulously annotated with bounding boxes for relevant table cells to ensure traceable and interpretable responses. EviFiVQA introduces new challenges for Large Language Models (LLMs) and Vision-Language Models (VLMs) by requiring multi-step numerical reasoning, explicit evidence localization, and hierarchical financial aggregation--all within complex, irregular table layouts. We benchmark state-of-the-art models on EviFiVQA, revealing their limitations in evidence-grounded understanding of financial statements. We release EviFiVQA to drive progress in audit-ready financial AI, enhancing transparency and reliability in AI-driven financial analytics. The dataset and code are available at https://github.com/sachinraja13/EviFiVQA