Abstract

Despite participants engaging in single modality stimuli, such as watching images or silent videos, recent work has demonstrated that multi-modal Transformer models can predict visual brain activity impressively well, even with incongruent modality representations. This raises the question of how accurately these multi-modal models can predict brain activity when participants are engaged in multi-modal stimuli. As these models grow increasingly popular, their use in studying neural activity provides insights into how our brains respond to such multi-modal naturalistic stimuli, i.e., where it separates and integrates information from different sensory modalities. We investigate this question by using multiple unimodal and two types of multi-modal models—cross-modal and jointly pretrained—to determine which type of models is more relevant to fMRI brain activity when participants were engaged in watching movies (videos with audio). We observe that both types of multi-modal models show improved alignment in several language and visual regions. This study also helps in identifying which brain regions process unimodal versus multi-modal information. We further investigate the impact of removal of unimodal features from multi-modal representations and find that there is additional information beyond the unimodal embeddings that is processed in the visual and language regions. Based on this investigation, we find that while for cross-modal models, their brain alignment is partially attributed to the video modality; for jointly pretrained models, it is partially attributed to both the video and audio modalities.

Abstract

LOCALITY and INTERFERENCE are two mechanisms which are attested to drive sentence comprehension. However, the relationship between them remains unclear—are they alternative explanations or do they operate independently? To answer this question, we test the hypothesis that in Hindi, interference effects (measured by semantic similarity and case markers) significantly predict locality effects (modelled using dependency length quantifying distance between syntactic heads and their dependents) within a sentence, while controlling for expectation-based measures and discourse givenness. Using data from the HindiUrdu Treebank corpus (HUTB), we validate the stated hypothesis. We demonstrate that sentences with longer dependency length consistently have semantically similar preverbal dependents, more case markers, greater syntactic surprisal, and violate intra-sentential givenness considerations. Overall, our findings point towards the conclusion that locality effects are reducible to broader memory interference effects rather than being distinct manifestations of locality in syntax. Finally, we discuss the implications of our findings for the theories of interference in comprehension.

Abstract

Methods and systems for obtaining end-to-end classification model using mixture density network

Abstract

Deep learning models are capable of complex auditory processing tasks such as keyword spotting, genre classification, and audio captioning, yet remain opaque. While several works have explored interpretability of neural networks for computer vision and natural language processing, the audio modality has been largely ignored. In this paper, we study the behavior of the audio CNN encoder used in the contrastively trained language-audio model, CLAP. In the domain of music and human speech sounds, we localize and identify the layers of the network that perform well on tasks of varying complexity, sometimes even outperforming the model's final outputs. Digging deeper, we also localize specific dataset classes to neuron clusters within a layer and analyze a cluster’s contribution to the model’s discriminability for that class. To perform these analyses, we propose an automated framework that can leverage a small dataset of a few thousand samples to evaluate and score neuron clusters for their role in classification. Our findings provide insights into the hierarchical nature of representations in audio CNNs, paving the way for improved interpretability of audio model.

Abstract

Air pollution, primarily driven by particulate matter (PM), significantly threatens public health. India, with three cities ranking among the world's top ten most polluted and with PM concentrations exceeding WHO guidelines by almost 11 times, urgent measures are needed to address this escalating crisis. AirIoT, a densely deployed IoT-based air quality monitoring network in Hyderabad, India, is an evidence-based approach to bringing awareness and increasing public participation by alleviating data scarcity.

Abstract

Emergency vehicles (EVs) perform a critical task of attending medical emergencies and delay in their operations can result in loss of lives to long term or permanent health implications. Therefore, it is very important to design strategies that can reduce the delay of EVs caused by slow moving traffic. Most of the existing work on this topic focuses on assignment and dispatch of EVs from different base stations to hospitals or finding the appropriate routes from dispatch location to hospital. However, these works ignore the effect of lane changes when EV is travelling on a stretch of a road. In this work, we focus on lane level dynamics for EV traversal and showcase that a pro-active picking of lanes can result in significant reductions in traversal time. In particular, we design a Reinforcement Learning (RL) model to compute the most optimal lane for an EV to travel at each timestep. We propose RLLS (Reinforcement Learning based Lane Search) algorithm for a general purposes EV trave

Abstract

In recent years, the rapid advancement of generative artificial intelligence (GenAI) has revolutionized the landscape of drug design, offering innovative solutions to potentially expedite the discovery of novel therapeutics. GenAI encompasses algorithms and models that autonomously create new data, including text, images, and molecules, often mirroring characteristics of existing datasets. This comprehensive review delves into the realm of GenAI for drug design, emphasizing recent advancements and methodologies that have propelled the field forward. Specifically, we focus on three prominent paradigms: transformers, diffusion models, and reinforcement learning algorithms, which have been exceptionally impactful in the last few years. By synthesizing insights from a myriad of studies and developments, we elucidate the potential of these approaches in accelerating the drug discovery process. Through a detailed analysis, we explore the current state and future directions of GenAI in the context of drug design, highlighting its transformative impact on pharmaceutical research and development.

Abstract

We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric video of skilled human activities (e.g., sports, music, dance, bike repair). 740 participants from 13 cities worldwide performed these activities in 123 different natural scene contexts, yielding long-form captures from 1 to 42 minutes each and 1,286 hours of video combined. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio, eye gaze, 3D point clouds, camera poses, IMU, and multiple paired language descriptions—including a novel “expert commentary” done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity, we also present a suite of benchmark tasks and their annotations, including fine-grained activity understanding, proficiency estimation, cross-view translation, and 3D hand/body pose. All resources are open sourced to fuel new research in the community.

Abstract

Curriculum learning has been used to improve the quality of text generation systems by ordering the training samples according to a particular schedule in various tasks. In the context of data-to-text generation (DTG), previous studies used various difficulty criteria to order the training samples for monolingual DTG. These criteria, however, do not generalize to the crosslingual variant of the problem and do not account for noisy data. We explore multiple criteria that can be used for improving the performance of cross-lingual DTG systems with noisy data using two curriculum schedules. Using the alignment score criterion for ordering samples and an annealing schedule to train the model, we show increase in BLEU score by up to 4 points, and improvements in faithfulness and coverage of generations by 5-15% on average across 11 Indian languages and English in 2 separate datasets. We make code and data publicly available

Abstract

We introduce a family of information leakage measures called maximal (α,β) -leakage ( Mα beL), parameterized by real numbers α and β greater than or equal to 1. The measure is formalized via an operational definition involving an adversary guessing an unknown (randomized) function of the data given the released data. We obtain a simplified computable expression for the measure and show that it satisfies several basic properties such as monotonicity in β for a fixed α , non-negativity, data processing inequalities, and additivity over independent releases. We highlight the relevance of this family by showing that it bridges several known leakage measures, including maximal α -leakage (β=1) , maximal leakage (α=∞,β=1) , local differential privacy (LDP) (α=∞,β=∞) , and local Rényi differential privacy (LRDP) (α=β) , thereby giving an operational interpretation to local Rényi differential privacy. We also study a conditional version of Mα beL on leveraging which we recover differential privacy and Rényi differential privacy. A new variant of LRDP, which we call maximal Rényi leakage, appears as a special case of Mα beL for α=∞ that smoothly tunes between maximal leakage ( β=1 ) and LDP ( β=∞ ). Finally, we show that a vector form of the maximal Rényi leakage relaxes differential privacy under Gaussian and Laplacian mechanisms.