Abstract

Videos have become an integral part of our daily digital consumption. With the widespread adoption of mobile devices, internet connectivity, and social media platforms, the number of online users and consumers has risen exponentially in recent years. This has led to an unprecedented surge in video content consumption and creation, ranging from short-form content on TikTok to educational material on Coursera and entertainment videos on YouTube. Consequently, there is an urgent need to study videos as a modality in Computer Vision, as it can enable a multitude of applications across various domains, including virtual reality, education, and entertainment. By understanding the intricacies of video content, we can unlock its potential and leverage its benefits to enhance user experiences and create innovative solutions. Producing video content at scale can be challenging due to various practical issues. The recording process can take several hours of practice, and setting up the right studio and camera equipment can be time-consuming and expensive. Moreover, recording requires manual effort, and any mistakes made during the shoot can be difficult to rectify or modify, often requiring the entire video to be re-shot. In this thesis, we aim to ask the question “Can synthetically generated videos take the place of real videos?” as automatic content creation can significantly scale digital media production and ease the process of content creation that can aid several applications. A form of human-centric representation that is becoming increasingly popular in the research community is the ability to generate talking-head videos automatically. Talking-head generation refers to the ability to generate realistic videos of a person speaking, where the generated video can be of a person that may not exist in reality or may exhibit significantly different characteristics than the original person. Recent deep learning approaches can synthesize synthetic talking-head videos at tremendous scale and quality, with diverse content and styles, that are visually indistinguishable from real videos. Therefore, it is imperative to study the process of generating talking-head videos as these videos can be used for a variety of applications, such as video conferencing, movie-making, broadcasting news, vlogging, and language learning among others. Consider a digital avatar reading news from a text transcript being broadcasted on news. In this vein, this thesis aims to explore two prominent use cases of generating synthetic talkingheads automatically - the first one towards generating large-scale synthetic content to aid people in lipreading at scale. The second use case is for automating the task of actor-double face-swapping in the moviemaking industry. We study and elucidate the challenges and limitations of the existing approaches, propose solutions based on synthetic talking head generation, and show the superiority of our methods through extensive experimental evaluation and user studies. In the first task, we address the challenges associated with learning to lipread. Lipreading is a primary mode of communication for people suffering from some form of hearing loss. Therefore, learning to lipread is an important aspect for hard-of-hearing people. However, learning to lipread is not an easy task and finding resources to improve one’s lipreading skills can be challenging. Existing lipreading training websites that provide basic online resources to improve lipreading skills, are unfortunately, limited by real-world variations in the talking faces, cover only a limited vocabulary, and are available in a few select languages and accents. This leaves the vast majority of users without access to adequate lipreading training resources. To address this challenge, we propose an end-to-end pipeline to develop an online lipreading training platform using state-of-the-art talking head video generator networks, textto-speech models, and computer vision techniques, to increase the amount of online content on the LRT platforms in an automated and cost-effective manner. We show that incorporating existing talking heading generator networks for the task of lipreading is not trivial, and requires careful adaptation. For instance, we develop an audio-video alignment module that aligns the speech utterance on the region with the mouth movements and adds silence around the aligned utterance. Such modifications are necessary to generate realistic-looking videos that don’t cause distress to the lipreaders. We also design carefully thought out lipreading training exercises, conduct extensive user studies, and perform statistical analysis to show the effectiveness of the generated content in replacing the manually recorded lipreading training videos. In the second problem, we address challenges in the entertainment industry. Body doubles play an indispensable role in the moviemaking industry. They take

Abstract

Word problem Solving is a popular NLP task that deals with solving mathematical problems described in natural language. Mathematical Word Problems cover problems over a large mathematical domain with various complexities ranging from Arithmetic and Algebraic to Geometry and Calculus. While most word problems are entirely textual, some word problems, like geometric word problems, may also have a visual component. Much research has been carried out to solve different genres of word problems with various complex- ity levels in recent years. However, most publicly available datasets and work are in English. Recently there has been a surge in word problem-solving in Chinese with the creation of large benchmark datasets. Labelled benchmark datasets for low-resource languages are very scarce. The first requirement for solving word problems, like any other problem, is data. To the best of our knowledge, no datasets are available for any Indian Languages for Word Problem Solving. Such limitations on data availability not only encouraged us to create a new dataset for an Indian Language but also made us explore techniques by which data for other Indian Languages can be created with ease. In this work, we present a diverse dataset containing 2336 Arithmetic word problems in Hindi built by manually crafting word problems and using word problems augmented from benchmark datasets of other languages. For augmentation, we used translation of word problems (of other languages in which Word Problem Solving data was developed) as a tool to generate diverse word problems. In this process, we studied the translated word problems, gathered the patterns of issues seen in these translations and defined the steps to eliminate the errors and improve the quality of the translated output for it to be suited to be a set of Hindi word problems that look as natural as word problems that are studied across India in Hindi medium schools. We also developed baseline systems for solving these word problems - a rule-based solver that uses verbs to identify operations for generating the answers to word problems and an end-to-end deep learning-based solver that generates equations for word problems. We also propose a new evaluation technique for word problem solvers taking equation equivalence into account. This will form the basis for future work for Word Problem Solving in Indian Languages, especially in Hindi.

Abstract

Autonomous driving requires accurate reasoning of the location of objects from raw sensor data. Recent end-to-end learning methods go from raw sensor data to a trajectory output via Bird’s Eye View (BEV) segmentation as an interpretable intermediate representation. Motion planning over cost maps generated via Birds Eye View (BEV) segmentation has emerged as a prominent approach in autonomous driving. However, current approaches have two critical gaps. First, the optimization process is simplistic and involves just evaluating a fixed set of trajectories over the cost map, which are not adapted based on their associated cost values. Second, the existing cost maps do not account for the uncertainty arising from noise in RGB images, BEV annotations. As a result, these approaches can struggle in challenging scenarios where there is abrupt cut-in, stopping, overtaking, and merging from neighboring vehicles. In this thesis, we propose UAP-BEV, a novel approach that models the noise in Spatio-Temporal BEV predictions to create uncertainty-aware occupancy grid maps. Using queries of the distance to the closest occupied cell, we obtain a sample estimate of the collision probability of the ego-vehicle. Subsequently, our approach uses gradient-free sampling-based optimization to compute low-cost trajectories over the cost map. Notably, the sampling distribution is adapted based on the optimal cost values of the sampled trajectories. By explicitly modeling probabilistic collision avoidance in the BEV space, our approach can outperform the cost-map-based baselines in collision avoidance, route completion, time to completion, and smoothness. To further validate our method, we also show results on the real-world dataset NuScenes, where we report collision avoidance and smoothness improvements. It also outperforms other uncertainty-based methods in conservatism.

Abstract

The ability to accurately identify the direction-of-arrival (DOA) and other signal parameters of an incoming radio signal is crucial in many applications, such as military and commercial communication systems. To meet this need, this paper proposes a system for direction finding and pulse parameter estimation that has been designed and implemented using a novel real-time multi-channel parallel Fast Fourier Transform (FFT) on a Field Programmable Gate Array (FPGA). The proposed system employs a unique approach that allows for the implementation of the FFT operation 5.5 times faster than the traditional IP-core-based implementation, with fewer arithmetic operations and a throughput of 1.35 Giga samples per second (Gsps). The parallel processing capabilities of the FPGA are utilized to enable the system to process multiple signals simultaneously in real-time. To achieve high precision in DOA estimation, the system employs the sum-difference monopulse technique. This technique utilizes the difference in the amplitudes of two signals received by two antennas. By comparing the amplitude difference with the sum of the signals, the angle of arrival of the incoming radio signal can be estimated with high precision, up to 1 degree . The proposed system’s hardware implementation has been thoroughly studied and verified experimentally through the development of a real-time prototype of the DOA estimation system. The prototype is capable of processing multiple signals in real-time and accurately estimating the direction-of-arrival of the incoming radio signal. Overall, the proposed system provides an efficient and reliable solution for direction finding and pulse parameter estimation in various commercial and military applications. The use of a novel realtime multi-channel parallel FFT on an FPGA enables fast and accurate signal processing, while the sum-difference monopulse technique ensures high precision in DOA estimation. This makes the system suitable for use in various fields, including communication systems, radar systems, and electronic warfare systems.

Abstract

The Standard Model of particle physics explains the particle content of matter and their interactions at the fundamental level. While the theory predicts and explains most subatomic events quite accurately, it has some limitations; also, some anomalies have been observed. Consequently, various theories beyond the standard model have been developed. Leptoquarks (LQs) and Right-handed Neutrinos (RHNs) are two hypothetical particles appearing in many of these new theories. They are well-known for their potential to address multiple issues. Experiments at the Large Hadron Collider (LHC) have put bounds on their parameters (if they exist) with certain assumptions. However, no direct experimental constraints exist on Leptoquark (LQ) couplings with quarks and right-handed neutrinos (RHNs) yet. If the RHNs are lighter than the LQs, they can be produced from the LQ decays. This simultaneous probe for RHNs and the LQs that dominantly couple to RHNs has never been searched for in experiments. In this thesis, we investigate the signals arising from the decays of Right-Handed Neutrinos (RHNs) coming from the decays of a scalar Leptoquark (LQ) that dominantly couples to RHNs. Our investigation covers both pair and single productions of the LQ and also RHN pair production through t-channel LQ exchange. These typically lead to two RHNs which can subsequently decay to multiple charged leptons and jets. While the dilepton final state is the most promising, our analysis centres on the more challenging monolepton final state to compute the discovery and exclusion reach of LQs and RHNs at the LHC. Here, we consider a scalar Leptoquark with an electromagnetic charge of either 1/3 or 2/3 (referred to as ϕ1 and ϕ2 respectively), primarily coupling to second-generation quarks and RHNs. We discuss the Standard Model and some Beyond Standard Model theories in chapter 1. We discuss particle accelerators and their simulation software in chapter 2. We analyse the monolepton final state to estimate the discovery reach at the high-luminosity LHC in chapter 3. In chapter 4, we visualise the distributions of various parameters and then look at them collectively to maximise the discovery reach. We present the results of the work in chapter 5. In chapter 6, we digress from the main topic and study a computation technique that can be used to automatically compute the bounds on LQ parameters from the current LHC data. We discuss an alternative methodology to compute the test on a contingency table where the table elements involve couplings as variables. These individual elements are summed up to form polynomials, and the polynomial form of χ2is computed. This allows for using computational techniques (like gradient descent) that cannot be utilised if the value of χ2 is computed using the regular tabular method at discrete values of the couplings.

Abstract

With the increased data availability, the biggest problem is getting accurate labeled data for training models. However, most of the time, we get the missing values. It is preferred to consider weak supervised learning methods to avoid the cost of quality labeled data. One such technique is called partial labeled data classification. We present a novel second-order cone programming framework to create robust classifiers that can tolerate uncertainty in the observations of partially labeled multiclass classification problems. Only the presence of second-order moments is necessary for the proposed formulations independent of the underlying distribution. The case of missing values in observations for multiclass classification problems is thus specifically addressed by these formulations. Adversarial examples are machine learning model inputs that an attacker has purposefully constructed to cause the model to make a mistake. Recent research aimed to improve the computational efficiency of adversarial training for deep learning models. Projected Gradient Descent (PGD) and Fast Gradient Sign Method (FGSM) are popular current techniques for generating adversarial examples efficiently. There is a trade-off between these two regarding robustness or training time. Adversarial training with the PGD is considered as one of the most efficient adversarial defense techniques to achieve moderate adversarial robustness. However, PGD requires high training time since it takes multiple iterations to generate perturbations. On the other hand, adversarial training with the FGSM takes much less training time since it takes one step to generate perturbations but fails to increase adversarial robustness. On the CIFAR-10/100 datasets, our approach outperforms PGD strong adversarial training techniques in terms of robustness to adversarial training and speed. While deep learning systems trained on medical images have demonstrated cutting-edge performance in various clinical prediction tasks, recent research indicates that cleverly created hostile images can fool these systems. Deep learning-based medical image classification algorithms have been questioned regarding their practical deployment. To attack this problem, we provide an unsupervised learning technique for detecting adversarial attacks on medical images. Without identifying the attackers or reducing classification performance, our suggested strategy FNS (Features Normalization and Standardization), can detect adversarial attacks more effectively than earlier methods.

Abstract

Neural Networks are known to be black box models which are explained by interpretability based approaches. ML Interpretability methods to explain the complex Neural Networks reasoning in human understandable form. Humans think in abstract concepts like color, texture, shapes, etc. Explaining black box models in these simple concepts aids human understanding of models leading to more transparency, reliability and proactive identification of risks/biases in the models. The recent interpretability based methods have started using concepts for explaining complex models in simple human understandable terms. In computer vision, concepts are defined as a set of images having a human-understandable meaning associated with them (eg. striped images form stripiness concept). Current concept based interpretability methods use the encoding of concepts in an intermediate layer of the model to define concept representations. They further use these representations by perturbing model activations in the intermediate layer and gauging its sensitivity to the perturbations. We use these CAV based sensitivity calculations for evaluation of desired properties by using the very definitions from the problem as concepts. We further extend the use of CAVs from post-hoc analysis to ante-hoc training via our novel concept loss and distillation paradigm. Concretely, we present a concept sensitivity based method to measure disentanglement in an ill-posed ambiguous problem of Intrinsic Image Decomposition (IID) followed by a novel method for concept based training of DNNs which utilizes conceptual knowledge from large pre-trained models in a distillation paradigm. IID involves decomposing an image into its constituent Reflectance and Shading components, which are illumination-invariant and albedo-invariant, respectively. We use this definition of IID to define our concepts and propose to use the sensitivity scores to directly measure the alignment of models predictions with the definition. For this, we measure the illumination invariance of Reflectance prediction and albedo invariance of Shading prediction by gazing model’s sensitivity to relevant concepts. We thus define the evaluation of IID in abstract human centered concepts. We define Concept Sensitivity Metric (CSM) to measure the disentanglement of Reflectance Shading in the models predictions for evaluating IID methods. We evaluate and interpret three recent IID methods on our synthetic benchmark of controlled albedo and illumination invariance sets. We also compare our metric with existing IID evaluation metrics on both natural and synthetic scenes and report our observations. Our metric not only surpasses several limitations of the existing metrics but is also consistent in both synthetic and real-world datasets. The concept based interpretability methods are post-hoc (after training) and can be used for analyzing the models. We aim to use the feedback provided by these post-hoc methods to train the model for further improvements in an ante-hoc manner. For this we propose a novel concept loss based on the CAV sensitivity. We argue that CAV learning in same model is not efficient and propose to use a separate model having knowledge of concepts in a knowledge distillation paradigm. We present Concept Distillation: a novel method for concept sensitive training of Deep Neural Networks. Concept Distillation can be used to sensitize or desensitize the student model towards user desired concepts. We show applications of our concept-sensitivity based training in debiasing in classification problems and prior induction in IID. We also introduce the TextureMNIST dataset to evaluate the presence of complex texture biases. Our concept-sensitive training can improve model interpretability, reduce biases, and induce prior knowledge.

Abstract

In AI, the ability of intelligent agent to model human player in games such as Backgammon, Chess and Go has been an important metric in benchmarking progress. Fundamentally, the games mentioned above can be characterized as competitive and zero-sum. In contrast, games such as Pictionary and Dumb Charades falls into the category of ‘social’ games. Unlike competitive games, the emphasis is on cooperative and co-adaptive game-play in a relaxed setting. Such social games can form the basis for the next wave of game-driven progress in AI. Pictionary™ is a wonderful example of cooperative game play to achieve a shared goal in communication-restricted settings. This popular sketch-based guessing game, which we employ as a use case, provides an opportunity to analyze shared goal cooperative game play in restricted communication settings. To enable the study of Pictionary and to understand various aspects associated with the game play, we designed a software ecosystem for web-based online game of Pictionary dubbed PICTGUESS. To overcome several technological and logistic barriers, which the actual game presents, we implemented a simplified setting for PICTGUESS wherein a game consists of a time-limited episode involving two players - a Drawer and a Guesser. The Drawer is tasked with conveying a given target phrase to a counterpart Guesser by sketching on a whiteboard within that time limit. However, occasionally some players in Pictionary draw atypical sketch content. While such content is occasionally relevant in the game context, it sometimes represents a rule violation and impairs the game experience. To address such situations in a timely and scalable manner, we introduce DRAWMON, a novel distributed framework for automatic detection of atypical sketch content in concurrently occurring Pictionary game sessions. We build specialized online interfaces to annotate atypical sketch content, resulting in ATYPICT, the first ever atypical sketch content dataset. We use ATYPICT to train CANVASNET, a deep neural atypical content detection network. We utilize CANVASNET as a core component of DRAWMON. Our analysis of post deployment game session data indicates DRAWMON’s effectiveness for scalable monitoring and atypical sketch content detection. Beyond Pictionary, our contributions can also serve as a design guide for customized atypical content response systems involving shared and interactive whiteboards.

Abstract

River water quality is a vital component of the ecosystem which is the main source for drinking, irrigation and more. In recent years, increasing urban growth and expansion have resulted in structural deterioration and functional degradation of river systems, resulting in significant volumes of harmful contaminants being discharged into the water. River pollution substantially impedes long-term economic and social growth and endangers human health. In order to enhance the water environment, it is essential to examine the reasons for water quality deterioration. River water quality modeling is the finest way to assess and monitor water quality. River water quality models are mathematical models that simulate the behavior of pollutants in surface water bodies used to predict the concentration of pollutants in the water and assess the overall river water quality. QUAL2K is a widely-used water quality model that can simulate water quality parameters, such as Dissolved Oxygen (DO), nutrients, and pH, in streams and rivers. However, simulation of many water quality parameters through water quality model does not assess the status and overall pollution extent of any river system. Single river water quality index is required for evaluation of overall pollution extent and to adopt necessary water quality control measures. Therefore, a holistic approach which can integrate river water quality simulation model and pollution extent characterization model is necessary. Thus, this study aims to develop an integrated, holistic model approach to estimate the pollution extent in terms of river water quality index utilizing various river water quality model parameters. Furthermore, the study aimed to analyze the spatial variability of river pollution extent using Geographic Information System. In this present study, the river water quality model, QUAL2K, is used to assess the quality of Bhadra river stretch, one of the major tributaries of the Tunga-Bhadra River situated in Karnataka. The study stretch considered is around 27 km which is divided into three reaches with elements of 1km as 3,4,20 for each reach respectively. In this current study, we analysed the effects of wastewater discharge from the monitoring stations such as the Mysore Paper Mill (MPM), the Visveshvarya Industrial Steel Limited (VISL), and the Bhadravathi city to simulate the Dissolved Oxygen by varying the Biochemical Oxygen Demand (BOD) loads (25%, 50%,70%,100%) coming from different pollutant sources within the study stretch. The observed water quality parameter data from 2006 to 2017 for those monitoring stations has been obtained from Advanced Centre for Integrated Water Resource (ACIWR). The water quality parameters like flow rate and pollution point sources discharge are the highly sensitive water quality parameters for modelling the QUAL2K model. There must be a reduction of 25% of BOD effluent to reach the minimum standards set by the Central Pollution Control Board (CPCB). It is noted that a 75% reduction of BOD effluent from point sources will lead to an increase of 15% average DO throughout the study stretch. The weighted Average Water Quality Index (WAWQI) method is used in estimating Water Quality Index (WQI) by using QUAL2K simulation results for the study stretch. QUAL2K model was calibrated from April 2006 to October 2013 and validated from November 2013 to March 2017. The estimated WQI values range from 92.35 to 112, indicating the quality classification ranges from very poor to unfit for consumption for the Bhadra river. It was observed that the quality status of Bhadra river water was very poor in upstream and downstream segments, while unfit for consumption in the middle segment indicating access to industrial and anthropogenic activities. For spatial analysis, Arc-Geographical Informal System (GIS) software and Inverse Distance Weighted (IDW) interpolation technique is used to generate water quality parameters concentration level and WQI maps. In context to this, it was found that the middle segment of Bhadra study stretch falls into the category of unfit for consumption due to the combination of effluents from industries like MPM and VISL and domestic effluents from Bhadravathi city. Overall, this study revealed that the MPM water quality station was a hotspot for river water quality degradation. Based on all these analyses, we provide a framework for the integration of the three components, i.e., QUAL2K calibration, WQI and spatial visualization for water quality management and policymakers to take decisions about water quality management in Bhadra river stretch .

Abstract

In this work, we develop three (related) schemes to generate novel molecules based on a seed molecule using an attentive captioning network. Input is the grid representation of the seed molecule, which could be one of the following: voxelised grid of the seed molecule, a grid reconstructed grid from it, or a sampled grid from it. The reconstructed grid is generated by passing the voxelised grid to the variational autoencoder and the sampled grid is generated by conditioning the decoder phase of the variational autoencoder on pharmacophoric requirements. The first scheme called ‘Direct Generation’ uses a RNN with the voxelised grid of the seed molecule as input to give an SMILES output of generated molecule. The second and third schemes utilize an additional variational autoencoder (VAE) to generate the grid which is used as input to RNN mentioned above in the subsequent step; the latent space of this VAE is modelled as a Riemannian manifold attached with a metric which is learnt along with the encoder and decoder networks of this VAE, which we name RHVAE. The second scheme, named ‘Autoencoded Generation’, takes as input the seed molecules’ voxelised grid representation for the encoder and its output along with the Remianian metric generates the latent space representation; this latent space representation along with the pharmacophoric requirement conditions form inputs for the decoder which outputs the ‘reconstruction grid’. In the third scheme, named ‘Sampled Generation’, starts with a point (and its Remanian metric) randomly sampled from latent space distribution learnt during training. This is evolved using either Hamiltonain Monte Carlo or Random Walk Monte Carlo, and then the evolved point with pharmacophoric conditions is sent to decoder to generate a ‘sampled grid’. In the subsequent step of Autoencoded Generation (Sampled Generation), this reconstructed grid (sampled grid) is sent to RNN for obtaining the SMILES of generated molecules. Overall, we demonstrate the generation of meaningful ligand shapes through the autoencoder network which can then be passed to our attentive captioning network to generate novel molecules while requiring smaller data sets for training while retaining the similar performance.