Abstract

Think of a situation where you put yourself in the shoes of a visually impaired person who wants to buy an item from a store or a person who is sitting in their house and watching the news on the television and wants to know about the content of the news being broadcast. Motivated by many more such situations where creating systems capable of understanding and reasoning over textual content in the videos, in this thesis, we tackle the novel problem of text-based video question answering. Vision and Language are broadly regarded as cornerstones of intelligence. Though each of these has different aims – language has the purpose of communication, and transmission of information, and vision has the purpose of constructing mental representations of the scene around us to navigate and interact with objects. When we study both of these fields jointly, it can result in applications, tasks, and methods that, when combined go beyond the scope compared to when they are used individually. This inter-dependency is being studied as a newly emerging area of a study named “multi-modal understanding”. Many tasks such as image captioning, visual question answering, video question answering, text-video retrieval, and more fall under the category of multi-modal understanding and reasoning tasks. To have a system that can reason over both text-based information and temporal-based information, we propose a new task. The first portion of this thesis focuses on the formulation of the text-based VideoQA task, by first analyzing the current datasets and works and thereby arriving at the need for text-based VideoQA. To this end, we propose the NewsVideoQA dataset where the question-answer pairs are framed on the text present in the news videos. As this is a new task proposed, we experiment with existing methods such as text-only models, single-image scene text-based models, and video question-answering models. As these baseline methods were not originally designed for the task of video question-answering using text in the videos, the need for a video question-answering model that can take the text in the videos into account to obtain answers became the need. To this end, we repurpose the existing VideoQA model to incorporate OCR tokens namely – OCR-aware SINGULARITY, a video question-answering framework that learns joint representations of videos and OCR tokens at the pretraining stage and also uses the OCR tokens at the finetuning stage. In this second portion of the thesis, we look into the M4-ViteVQA dataset which aims to solve the same task of text-based video question-answering but the videos belong to multiple categories such as shopping, traveling, vlogging, gaming, and so on. We perform a data exploratory analysis where we analyze both NewsVideoQA and M4-ViteVQA on several aspects that look for limitations in these datasets. Through the data exploratory experiment, we show that most of the questions in both datasets have questions that can be answered just by reading the text present in the videos. We also observe that most of the questions can be answered using a single to few frames in the videos. We perform an exhaustive analysis on a text-only model: BERT-QA which obtains comparable results to the multimodal methods. We also perform cross-domain experiments to check if training followed by finetuning on two different categories of videos helps the target dataset. In the end, we also provide some insights into creating a dataset and how certain types of annotations can help the community come up with better datasets in the future. We hope this work motivates future research on text-based video question-answering in multiple video categories. Furthermore, the pretraining strategies and combined representation learning from these videos and the multiple modalities that videos provide us will help create scalable systems and drive future research towards better datasets and creative solutions.

Abstract

Wikipedia is one of the primary sources of encyclopedic content online. It is one of the most widely read websites. It is also regarded as a quality data source in many machine-learning pipelines. To maintain the high quality of its articles, Wikipedia has three core content policies, one of which is “Neutral Point of View (NPOV)”. This policy is a set of principles, including “avoiding stating opinions as facts” and “preferring nonjudgmental language.” Whenever we refer to “bias”, we refer to it within these guidelines. This work studies how to enhance the quality of the Indian language Wikipedia articles. We looked at existing work on dataset curation from English Wikipedia for bias detection and tried replicating that for Indian languages. We discuss the hurdles faced in this process and discuss translation (along with various quality checks to reduce noise) as a viable alternative. Much of this thesis is dedicated to automatically detecting whether a sentence can be called biased and trying to remove the bias if so. Bias detection is challenging because certain words lead to bias if written in some contexts while not in others. For bias detection in Indian languages, we perform binary classification using MuRIL, InfoXLM and mDeBERTa in zero-shot, monolingual and multilingual settings. For human evaluation, we note how this is a subjective task and disagreement among annotators is expected. Thus, we also experiment with different settings like loss functions specific for subjective tasks and include anonymized annotator-specific information to help us understand the level of disagreement. For bias mitigation, we perform style transfer using IndicBART, mT0 and mT5. These models provide strong baseline results for the novel multilingual tasks. We study how different text generation metrics may or may not be able to capture the quality of debiasing and how to evaluate our models best. Reinforcement learning offers a way to fix the problems observed in the debiased results of the style transfer module. Also, it helps us combine the classification and style transfer modules. We formulate three reward functions specific to our debiasing task and study the results of training fully/partially with these rewards compared to vanilla mT5. Yet another way of improving the quality of the Indian language Wikipedias is to verify the accuracy and reliability of the information presented. All material in Wikipedia must be attributable to a reliable, published source. Thus, we try to identify if the information in a sentence is factually correct or needs a citation. In contrast to previous work, we do this at the fact level instead of the sentence level for more accurate results. Thus, these measures will enhance the quality of the Indian language Wikipedia articles and increase its credibility as the largest source of free, fair, and accurate information.

Abstract

Consider an autonomous agent capable of obeying the following instruction �Go and clean the coffee spilled on the dining table�. To perfectly execute this instruction, the agent needs to have a precise understanding of its dynamic 3D environment. The final task can be broken down into the following sub-tasks; 3D grounding, 3D semantic understanding, and 3D motion planning. To excel in all these tasks, 3D representation learning plays an important role. Motivated to contribute towards creating systems that can perceive and act in real 3D words, in this thesis, we propose two novel methods for 3D representation learning. The first portion of this thesis focuses on the tasks of unsupervised domain adaptation (UDA) for 3D point clouds. The point cloud data acquisition procedures manifest themselves as significant domain discrepancies and geometric variations among both similar and dissimilar classes. The standard domain adaptation methods developed for images do not directly translate to point cloud data because of their complex geometric nature. Existing works mainly focused on designing a self-supervised task to improve adaptation performance. We propose a new UDA architecture for point cloud classification that benefits from multimodal contrastive learning to get better class separation in both domains individually. Further, the use of optimal transport aims at learning source and target data distributions jointly to reduce the cross-domain shift and provide a better alignment. We conduct a comprehensive empirical study on PointDA 10 and GraspNetPC-10 and show that our method achieves state-of-the-art performance on GraspNetPC-10 (with ? 4-12% margin) and best average performance on PointDA-10. Our ablation studies and decision boundary analysis also validate the significance of our contrastive learning module and OT alignment. In the second portion of the thesis, we explore learning Wasserstein embeddings for point clouds towards multiple downstream tasks. As learning embeddings of any data largely depends on the ability of the target space, we propose to embed point clouds as discrete probability distributions in Wasserstein space. We build a contrastive learning setup to learn Wasserstein embeddings that can be used as a pre-training method with or without supervision towards any downstream task. We show that the features captured by Wasserstein embeddings are better in preserving the point cloud geometry, including both global and local information, thus resulting in improved quality embeddings. We perform exhaustive experiments and demonstrate the effectiveness of our method for point cloud classification, transfer learning, segmentation, and interpolation tasks over multiple datasets, including synthetic and real-world objects. We also compare against recent methods that use Wasserstein space and show that our method outperforms them in all downstream tasks. Additionally, our study reveals a promising interpretation of capturing critical points of point clouds that makes our proposed method self-explainable. We hope this work motivates future research in utilizing optimal transport for understanding the real 3D world. We also hope that the self-supervised approaches proposed in this thesis will act as a step towards in this direction.

Abstract

The convergence of Deep Reinforcement Learning (DRL) and cognitive neuroscience has opened avenues for probing neural representations within the brain. This thesis delves into the synergy between decision-making and representation by harnessing DRL’s capabilities. Through a focus on dot motion perceptual decision-making task within a high-dimensional framework akin to psychological experiments, we acquire novel insights into how intricate neural networks tackle complex tasks. The resultant end-to-end model not only elucidates network strategies, but also offers a backdrop for understanding analogous brain processes. Investigation reveals intriguing resemblances between the network’s behavior and neural mechanisms observed in the middle temporal visual area (MT) of the brain, known for encoding direction and motion strength. Remarkably, direction selectivity emerges solely through reward-driven training, while graded firing patterns encode motion strength. A testable hypothesis arises, suggesting coherenceselective neurons within the MT population. While conventional Reinforcement Learning (RL) presupposes uniform data distributions, real-world scenarios like autonomous driving and natural environments display Zipfian distributions, characterized by frequent occurrences amidst rare events. Inspired by complementary learning systems theory, this work introduces an architecture tailored for learning from Zipfian distributions. Unsupervised discovery of long tail states and their retention in episodic memory, coupled with recurrent activations, forms the core of the proposed architecture. Retrieval from episodic memory via similarity-based search, reinforced with weighted importance, amplifies performance across diverse Zipfian tasks. Our proposed architecture achieves higher accuracy than the IMPALA algorithm across all three tasks and evaluation metrics (Zipfian, Uniform, and Rare Accuracy). This thesis advances the nexus of decision-making and neural representation through DRL, while proposing an innovative architecture capable of navigating the intricacies of Zipfian data distributions. It not only augments our grasp of cognitive building blocks but also bears implications for resolving challenges across real-world domains. This research in one small step that bridges the gap between computational models and cognitive processes, opening new vistas for understanding and application.

Abstract

Insurance plays a vital role in safeguarding individuals and businesses against various risks. Understanding the factors that influence insurance consumption is crucial for policymakers, insurance companies, and consumers alike. This thesis examines various aspects of insurance demand, focusing on understanding consumer behaviour related to insurance purchase decisions. Two studies were conducted in this work. The first study investigated the influence of wealth level on insurance consumption behavior using a game application. The results showed a nonlinear correlation between wealth and insurance consumption, increasing with increasing wealth and achieving a negative correlation after a threshold. The second study used a game-like application to determine how various behavioral characteristics, such as risk tolerance and loss framing, affect the decision to purchase insurance. The study focused on university students aged 18 to 25 with no income and working-class people aged 35 to 55. The results showed that the effect of loss framing on insurance consumption depends on the particular insurance policy, even if participants are aware of the loss. The studies explore data collection methods using surveys and game-like applications for studying insurance consumption behavior. Surveys can lead to dishonest responses, misunderstandings, and errors in interpretation. Gaming applications offer a more authentic and ecologically valid context, framing, capturing real-time behaviors and decision-making. These applications capture various aspects of insurance consumption behavior, such as decision-making processes, choices, and engagement patterns. The findings can be directly extrapolated or modeled to predict human behavior in real-life situations.

Abstract

In the realm of open-domain question answering, a common challenge arises from the inherent ambiguity of user queries. Conventional question-answering systems that offer a single answer often falter when confronted with ambiguity, as questions may be subject to multiple interpretations and yield various distinct answers. This paper addresses this challenge through the lens of multi-answer retrieval, a task focused on retrieving passages capable of capturing the diverse array of answers to a single question. Our approach introduces a re-ranking methodology that leverages Determinantal point processes, employing BERT embeddings as kernels. This technique takes a holistic view by jointly considering both query-passage relevance and passage-passage correlation. The goal is to retrieve passages that not only align with the user’s query but also encompass a diversity of information. Empirical results underscore the effectiveness of our re-ranking approach, demonstrating its superiority over state-of-the-art methods, particularly when evaluated on the AmbigQA dataset. In parallel, the field of question-answering, powered by reader models or answer generation, has seen significant advancements. Neural sequence-to-sequence models, especially those incorporating attention mechanisms, have played a vital role in this progress. However, these models encounter a limitation as they predominantly rely on pointer generators that exclusively target words from the source passage, even though their goal is to provide answers that involve both the question and the source text. To address this constraint, we introduce an innovative query pointer module within a comprehensive multipointer generator framework. This module facilitates the generation of answers that combine question and passage information in diverse ways, resulting in contextually relevant responses that effectively address the nuances of user queries. Our model demonstrates a significant improvement of 3.5 points in Rouge-L scores when compared to the state-of-the-art model on existing datasets. These advancements collectively signify substantial progress in both multi-answer retrieval and question-answering research, offering the promise of enhanced capabilities in information retrieval, natural language understanding, and content summarization.

Abstract

The utilization of quantum mechanical characteristics such as entanglement serves as a critical foundation in a multitude of quantum information processing tasks. With the objective of designing quantum networks, broadcasting - which is defined as the process of producing more quantum mechanical resources from a limited amount - presents an encouraging approach. One promising method of achieving broadcasting in quantum mechanical systems is through cloning operations. This thesis delves into the investigation of broadcasting of a class of quantum entangled states that go beyond 2 ⊗ 2 systems. It is known that beyond 2⊗2 and 2⊗3 dimensional quantum systems, Peres-Horodecki criterion is no longer sufficient to detect separability of quantum states. This is because there exists entangled states with both positive and negative partial transpose (PPT and NPT). Further, it is also true that all PPT entangled states are bound entangled states. However, in the class of NPT states, there can exist bound entangled states as well as free entangled states. All free/useful/distillable entanglement is a part of the class of NPT entangled states. In this thesis, the question is asked that given an NPT entangled state in 3 ⊗ 3 dimensional system as a resource, how much entanglement can we broadcast so that resource still remains NPT. The initial chapter offers a brief overview of the historical occurrences surrounding the evolution of Quantum mechanics. It is followed by a brief and succinct introduction to the necessary theory and mathematical tools required for this thesis. In the next chapter a 3 ⊗ 3 system is chosen as a first step to understand broadcasting of NPT states in higher dimensional systems. In particular, we find out the range of broadcasting of NPT entanglement for Two parameter Class of States (TPCS) and Isotropic States (IS). Interestingly, as a derivative of this process we are also able to locate the existence of absolute PPT states (ABPPT) in 3⊗3 dimensional system. Here we implement the strategy of broadcasting through approximate cloning operations

Abstract

The recent surge in healthcare sensing technologies has revolutionized the landscape of patient care and safety monitoring. This thesis delves into the burgeoning field of healthcare sensing and addresses the critical need for advancements in monitoring systems applicable to various domains, including elderly care, critical care units, and prolonged home monitoring. These applications necessitate the continuous and non-intrusive measurement of vital signs such as heart rate, blood-oxygen saturation, temperature, and subject movement. This research focuses on enhancing patient comfort and convenience by introducing innovative solutions. Firstly, the thesis introduces a receiver front-end integrated circuit (IC) tailored for radar systems enabling non-contact-based measurement of vital signs. This radar-based approach offers a non-invasive method for obtaining crucial health parameters, promising increased accuracy and reliability while minimizing patient discomfort. Furthermore, in response to the growing demand for portable safety systems in industrial settings, the thesis presents the development of a portable industry safety system. This system is designed to provide comprehensive safety monitoring in industrial environments, ensuring the well-being of workers through real-time monitoring and hazard detection. Through the exploration of these technological advancements, this thesis contributes to the field of healthcare sensing and safety systems. The proposed receiver front-end IC for radar-based vitals monitoring offers a novel approach to non-contact sensing, while the portable industry safety system aims to enhance workplace safety through continuous monitoring and hazard detection. Overall, these advancements pave the way for improved healthcare monitoring and industrial safety, addressing critical needs in both domains.

Abstract

In the field of aerial transport, specifically in operations involving quadrotors carrying suspended payloads, two critical stages are the attachment and detachment of the payload. These procedures are challenging due to various uncertainties inherent in the quadrotor, environmental variables, and payload swings caused by human or external interactions. These uncertainties can escalate quickly, posing significant risks to the quadrotor and payload and, crucially, to the human operator involved in the attachment/detachment process. Current advanced controllers in this field often need to adequately address these uncertainties, typically treating them as bounded, predictable variables. However, this approach must be revised to manage the unpredictable nature of payload swings during these critical stages. To overcome this limitation, our research introduces an innovative adaptive anti-swing controller. This controller uniquely considers uncertainties state-dependently, specifically tailored to address swing induced by the unpredictable aspects of attaching and detaching suspended payloads. In this work, we have conducted a thorough analytical evaluation of the closed-loop stability of this new system. Additionally, real-time experimental trials have been carried out. The results from these experiments demonstrate a marked performance improvement compared to existing state-of-the-art systems. This novel approach shows promise in enhancing safety and control effectiveness in aerial payload transportation, especially during the vital phases of payload and unclasping.

Abstract

Major goal of this thesis is to study the dialectal variations and improve the performance of speech recognition with an embeddings derived from improved dialect classification system. Initial studies focused on improvement of the dialect classification system with three major dialects (AU:Australian, UK:Britain, and US:American) of English. In order to improve the performance of dialect classification system and based on the analysis of dialectal variations, advanced signal processing approaches were proposed to investigate for dialect classification with traditional i-vector system. The features that provide high spectral resolution will help to capture subtle differences between dialects. So, this thesis proposed to use single frequency filtering (SFF) and zero-time windowing (ZTW) based features that provide high spectral resolution without compromising temporal resolution. Along with frame level spectral resolution, longer temporal context will constitute for dialect classification. So, approaches that enhance the temporal context of proposed features (SFF and ZTW) approaches such as delta and double delta coefficients (∆+∆∆), shifted delta coefficients (SDCs) are experimented. It is observed that dialect classification system has given promising performance with the proposed features with temporal context provided by ∆+∆∆ and SDCs. Further, signal processing approaches that can provide long temporal summarization such as frequency domain linear prediction (FDLP) are proposed for dialect classification. From experiments, with FDLP based features, it is observed that long temporal summarization provided by FDLP based features is advantageous for discriminating dialects. So, both the signal processing approaches that provide high spectral resolution (SFF and ZTW) and long temporal summarization (FDLP) have shown to give promising performance in dialect classification when compared to commonly used STFT based features. Further, due to promising performance by deep neural networks in classification tasks and its ability to provide longer temporal context, simpler (CNN) to advanced deep neural network (TCN, TDNN, and ECAPA-TDNN) architectures that provide different temporal contexts are investigated, it is observed that advanced neural network architectures improved the performance of dialect classification. Further, on evaluation of the best of both stages, it is observed that ECAPA-TDNN performed better with proposed features (SFF). The dialectal variations in speech degrade the performance of multi-dialectal automatic speech recognition (ASR) system. The embeddings derived from the best dialect classification system are applied to multi-dialect (with AU, UK, and US dialects) ASR and found to improve the performance of the ASR system. In most studies, Indian English is considered as a single dialect even though it has different native speakers. So, the inclusion of foreign dialectal embeddings improved the performance of the ASR system. The observations made in dialect classification systems with major dialects of English are extended to foreign dialect classification (i.e., native language (or L1) identification). The embeddings extracted from the improved dialect classification system are included along with the Indian English ASR system to improve the performance.