Abstract

Current Non-Audible Murmur (NAM)-to-speech techniques rely on voice cloning to simulate ground-truth speech from paired whispers. However, the simulated speech often lacks intelligibility and fails to generalize well across different speakers. To address this issue, we focus on learning phoneme-level alignments from paired whispers and text and employ a Textto-Speech (TTS) system to simulate the ground-truth. To reduce dependence on whispers, we learn phoneme alignments directly from NAMs, though the quality is constrained by the available training data. To further mitigate reliance on NAM/whisper data for ground-truth simulation, we propose incorporating the lip modality to infer speech and introduce a novel diffusionbased method that leverages recent advancements in lip-tospeech technology. Additionally, we release the MultiNAM dataset with over 7.96 hours of paired NAM, whisper, video, and text data from two speakers and benchmark all methods on this dataset. Speech samples and the dataset are available at https://diff-nam.github.io/DiffNAM/

Abstract

Embodied agents assisting humans are often asked to complete a new task in a new scenario. An agent preparing a particular dish in the kitchen based on a known recipe may be asked to prepare a new dish or to perform cleaning tasks in the storeroom. There may not be sufficient resources, e.g., time or labeled examples, to train the agent for these new situations. Large Language Models (LLMs) trained on considerable knowledge across many domains are able to predict a sequence of abstract actions for such new tasks and scenarios, although it may not be possible for the agent to execute this action sequence due to task-, agent-, or domain-specific constraints. Our framework addresses these challenges by leveraging the generic predictions provided by LLM and the prior domain-specific knowledge encoded in a Knowledge Graph (KG), enabling an agent to quickly adapt to new tasks and scenarios. The robot also solicits and uses human input as needed to refine its existing knowledge. Based on experimental evaluation over cooking and cleaning tasks in simulation domains, we demonstrate that the interplay between LLM, KG, and human input leads to substantial performance gains compared with just using the LLM output.

Abstract

We study the connection between audio-visual observations and the underlying physics of a mundane yet intriguing everyday activity: pouring liquids. Given only the sound of liquid pouring into a container, our objective is to automatically infer physical properties such as the liquid level, the shape and size of the container, the pouring rate and the time to fill. To this end, we: (i) show in theory that these properties can be determined from the fundamental frequency (pitch); (ii) train a pitch detection model with supervision from simulated data and visual data with a physics-inspired objective; (iii) introduce a new large dataset of real pouring videos for a systematic study; (iv) show that the trained model can indeed infer these physical properties for real data; and finally, (v) we demonstrate strong generalization to various container shapes, other datasets, and in-the-wild YouTube videos. Our work presents a keen understanding of a narrow yet rich problem at the intersection of acoustics, physics, and learning. It opens up applications to enhance multisensory perception in robotic pouring.

Abstract

Image captioning systems are unable to generate fine-grained captions as they are trained on data that is either noisy (alt-text) or generic (human annotations). This is further exacerbated by maximum likelihood training that encourages generation of frequently occurring phrases. Previous works have tried to address this limitation by fine-tuning captioners with a self-retrieval (SR) reward. However, we find that SR fine-tuning has a tendency to reduce caption faithfulness and even hallucinate. In this work, we circumvent this bottleneck by improving the MLE initialization of the captioning system and designing a curriculum for the SR fine-tuning process. To this extent, we present (1) Visual Caption Boosting, a novel framework to instill fine-grainedness in generic image captioning datasets while remaining anchored in human annotations; and (2) BagCurri, a carefully designed training curriculum that more optimally leverages the contrastive nature of the self-retrieval reward. Jointly, they enable the captioner to describe fine-grained aspects in the image while preserving faithfulness to ground-truth captions. Our approach outperforms previous work by +8.9% on SR against 99 random distractors (RD100) (Dessi et al., 2023); and +7.6% on ImageCoDe. Additionally, existing metrics to evaluate captioning systems fail to reward diversity or evaluate a model's fine-grained understanding ability. Our third contribution addresses this by proposing self-retrieval from the lens of evaluation. We introduce TrueMatch, a benchmark comprising bags of highly similar images that uses SR to assess the captioner's ability to capture subtle visual distinctions. We evaluate and compare several state-of-the-art open-source MLLMs on TrueMatch, and find that our SR approach outperforms them all by a significant margin (e.g. +4.8% - 7.1% over Cambrian) while having 1-2 orders of magnitude fewer parameters. We also outperform vanilla SR by +14.4% to +19.5%.

Abstract

This paper introduces ViNet-S, a 36MB model based on the ViNet architecture with a U-Net design, featuring a lightweight decoder that significantly reduces model size and parameters without compromising performance. Additionally, ViNet-A (148MB) incorporates spatio-temporal action localization (STAL) features, differing from traditional video saliency models that use action classification backbones. Our studies show that an ensemble of ViNet-S and ViNet-A, by averaging predicted saliency maps, achieves state-of-the-art performance on three visual-only and six audio-visual saliency datasets, outperforming transformer-based models in both parameter efficiency and real-time performance, with ViNet-S reaching over 1000fps.

Abstract

Automatic detection and severity assessment of dysarthria are crucial for delivering targeted therapeutic interventions to patients. While most existing research focuses primarily on speech modality, this study introduces a novel approach that leverages both speech and text modalities. By employing cross-attention mechanism, our method learns the acoustic and linguistic similarities between speech and text representations. This approach assesses specifically the pronunciation deviations across different severity levels, thereby enhancing the accuracy of dysarthric detection and severity assessment. All the experiments have been performed using UA-Speech dysarthric database. Improved accuracies of 99.53% and 93.20% in detection, and 98.12% and 51.97% for severity assessment have been achieved when speaker-dependent and speaker-independent, unseen and seen words settings are used. These findings suggest that by integrating text information, which provides a reference linguistic knowledge, a more robust framework has been developed for dysarthric detection and assessment, thereby potentially leading to more effective diagnoses.

Abstract

One-class classification (OCC) is a machine learning technique employed in scenarios involving imbalanced data, limited labeled data, or the need to detect anomalies or novel patterns. One-Class Classification by Ensembles of Regression (OCCER) models have shown excellent performance for OCC problems. In this paper, we study whether the combinations of ensemble methods such as random subspace and random projection with OCCER can be used to improve the performance of OCCER. Our main objective is to explore the impact of diversity creation methods, including bagging, random subspace, and random projections, on enhancing the performance of the OCC ensemble method. Additionally, we investigate the feasibility of applying the OCCER ensemble method to a novel low-dimensional space generated by autoencoders, all while maintaining high prediction accuracy. The experimental results over 37 datasets demonstrate that the creation of diverse OCCER models using random subspace and random projection substantially improves the performance of the OCCER. The diversity creation method, Random subspaces, outperforms other diversity creation methods.

Abstract

Autoencoders have been used for finding inter- pretable and disentangled features underlying neural network representations in both image and text domains. While the efficacy and pitfalls of such methods are well-studied in vision, there is a lack of corresponding results, both qualitative and quantitative, for the text domain. We aim to address this gap by training sparse autoencoders (SAEs) on a synthetic testbed of formal languages. Specifically, we train SAEs on the hidden representations of models trained on formal languages (Dyck-2, Expr, and English PCFG) under a wide variety of hyperparameter settings, finding interpretable latents often emerge in the features learned by our SAEs. However, similar to vision, we find performance turns out to be highly sensitive to inductive biases of the training pipeline. Moreover, we show latents correlating to certain features of the input do not always induce a causal impact on model’s computation. We thus ar- gue that causality has to become a central tar- get in SAE training: learning of causal features should be incentivized from the ground- up. Motivated by this, we propose and perform preliminary investigations for an approach that promotes learning of causally relevant features in our formal language setting.

Abstract

In the Himalayan region, traditional buildings made of masonry have existed for centuries and still make up a sizable portion of the building stock. Many of these buildings have proven records of earthquake resilience. At the same time, the primary cause of fatalities from previous earthquakes is the increasing collapse of unreinforced masonry structures. In various Himalayan locations, local seismic culture has evolved over the past few centuries to improve the seismic capacity of stone constructions. These areas use wood or other materials with high tensile strength to reinforce their stone masonry. The post-earthquake reconnaissance surveys clearly demonstrate the superior performance of these timber-reinforced masonry constructions. Nevertheless, there is still a lack of scientific investigation, either through numerical modelling or laboratory-based studies, on the behaviour and mechanism of these structures. Furthermore, there are no explicit criteria for evaluating timber-reinforced masonry in the present Indian seismic code. This study examines several distinct traditional masonry building typologies found in the Himalayas, such as ‘Thathara’, ‘Kath-Kuni’ or ‘Koti-Banal’, ‘Newari’, ‘Dhajji Dewari’, ‘Ikra’ or ‘Assam-Type House’, ‘Bhatar’ or ‘Taq’, and ‘Dry-stone buildings’. For the majority of these typologies, aside from Dry-stone houses, the masonry walls have embedded horizontal timber bands or lacings that enhance box behaviour, thereby preventing premature failure. Through a thorough analysis of timber-reinforced masonry, including architectural details, earthquake-resilient & vulnerable features, and experimental and numerical investigations, the article presents a unique addition to our understanding of seismic performance and how it might best protect the Himalayan region’s traditional heritage. Additionally, the current study contributes to a better understanding of how wood elements give masonry structures greater seismic resilience. This understanding will help with several initiatives: (i) better conservation, preservation, and maintenance of traditional or heritage structures; (ii) seismic codes/standards providing details regarding the assessment of timber-reinforced structures; (iii) supporting and encouraging practitioners to design and build using techniques that have demonstrated superior performance, and (iv) as attempts to lessen the impact of building on our environment increases, provide a way to make safer structures utilizing low-carbon materials.

Abstract

Indian raga music requires a symbolic notation dataset to support music information retrieval (MIR) tasks. With the growing applications of generative artificial intelligence, text-based datasets can be especially valuable, as these models excel when inputted structured symbolic data. A high-level analysis of ragas is difficult from audio data. In this article, we announce our new dataset called InRaMu. Notations of raga-based compositions in Hindustani, Carnatic and Rabindra Sangeet were scraped from the Internet. The compiled dataset consists of 7 sources combined to get 2,295 compositions in 337 ragas. Total number of compositions, with 976,826 individual notes, correspond to a combined playback duration of about 16,280 minutes. Special care was taken during data preprocessing such as filtering of compositions to ensure raga accuracy. This article provides the overview of dataset format, and provides statistical information, along with potential applications of the dataset.