Abstract

Image quality is a fundamental problem in computer vision. For variety of applications, for instance scanning documents, QR codes, bar codes or algorithms like object detection, recognition, tracking, scene understanding etc. images with good contrast, high illumination and sharpness are desired. Similarly in computer graphics for information visualisation, animations, presentations etc. aesthetically pleasing design and good colorization of the images are desired. Therefore the definition of image quality depend on the context and application of the image. In this thesis we attempt to address various challenges pertaining to image quality, (1) for natural imaging, we explore a novel approach for predicting the capture quality of the images taken in the wild. (2) For Graphics designs, we explore the aesthetic quality of images by suggesting multiple aesthetically pleasing colorization of graphics designs. Due to increasing advancements and portability of smartphone cameras, it has become a default choice for capturing images in the wild. However, there are quality issues with camera captured images due to reasons like lack of stability during capture process. This hinders the automatic workflows which takes camera captured images as input e.g. Optical Character Recognition (OCR) for documents image, face detection/recognition from human image etc. Part of this thesis is focused on Image Quality Assessment (IQA), the aim is to quantify the degradation like out-of-focus blur and motion artefacts in a given image. One of the major challenge in IQA for images captured in the wild is that, we do not have ground truth to measure the capture quality. Therefore various previous attempts of IQA require human in loop for creating the ground truths for capture quality of images. Large user studies are conducted and mean human opinion scores are then used as measure for quality. In this work we use a signal processing based technique to generate the IQA ground truth, and propose a comprehensive IQA dataset which is a good representative of the real degradation during the process of capture. Further, we propose deep learning based approach to predict image quality for captures in the wild. Such IQA algorithm can be helpful in the cause by either giving online quality suggestion during capture or rating the quality post capture. Another dimension to the image quality is aesthetic quality. Increasing usage of internet, social media and advancement in the mobile camera, photography has become a very popular hobby and interest to a large section. Even for a well captured image, people use varieties of filters and effects post capture to enhance the appearance of the image e.g. adjusting color temperature, contrast or even blurring part of vi vii the image (bokeh effect) etc. Therefore the capture quality is not enough to define the aesthetic quality of the image. However, one of the major factors defining the aesthetic quality of image is colorization. Particularly in computer graphics domain, where artificially generated images already have well defined structures, shape and components. Therefore sharpness or capture quality are not relevant, but on the other hand, color quality plays a very important role in visualization and appearance of the image. In natural images, largely colors are associated with semantics e.g. sky is always blue or grass are green etc. whereas in animations and computer graphics where objects are loosely associated with semantics, this lead to more choices of colors. Hence the problem of colorization becomes more challenging in graphics domain, where overall appearance of the images are more than its naturalness. Therefore, this work also covers aesthetic quality of graphics images, here instead of measuring the color quality, we propose algorithm to produce better coloring suggestions for the given graphics images.

Abstract

Dysarthria is a disorder resulting from weaknesses of neuromuscular execution in motor speech production due to brain tumors, brain injury, stroke, cerebral palsy, and facial paralysis. The abnormalities in the resonance, articulation, respiration, phonation, and prosody of speech are associated with dysarthria leads to poor speech intelligibility. The symptoms of poor speech quality and reduced intelligibility can be used to identify the dysarthria. The degree to which the listener understands the dysarthric individual’s speech is referred to as speech intelligibility of that speaker. Dysarthric speech detection and intelligibility assessment are very important steps in the clinical diagnosis of dysarthria. The subjective intelligibility assessment methods can be influenced by the listener’s familiarity with patients, the contextual, suprasegmental factors, and semantic/syntactic features. Moreover, the subjective intelligibility assessment methods are costly and time-consuming. On the other hand, objective intelligibility assessment methods are economical, repeatable, reliable, and can help remote patient rehabilitation monitoring. The growing evidence suggesting that clinicians are becoming more receptive to objective intelligibility assessment systems in which the trained acoustic model can assess the speech intelligibility. An objective assessment method like acoustic to articulatory representation of dysarthric speech can effectively uncover the dysarthria specific features that are useful for the assessment. Additionally, the detection of articulatory changes from the speech is useful for the assessment of dysarthria. Hence, the extraction of features from speech signals is considered to be one of the most important steps in developing clinical tools for the automatic detection and assessment of dysarthria. In literature, few attempts have been made to detect dysarthria, assess the intelligibility of dysarthric speech, and identify the type of dysarthria. The main objectives of this thesis are dysarthric speech detection (binary classification task) and dysarthric speech intelligibility assessment (four class classification task). In this thesis, all the studies are done using the UA-Speech database. In the first study, the effect of tongue tip movement in subjects with dysarthria was investigated by analyzing the duration of rhotic approximant sounds. The duration of rhotic approximant was measured using the third formant trajectory estimated from speech. The formant trajectory is estimated from the spectrogram computed using quasi-closed-phase (QCP) analysis that is an accurate method to estimate formants. This study showed that the duration of rhotic approximant is a good indicator of the severity level of dysarthria. This method requires sophisticated landmark detection techniques to measure the rhotic approximant duration for the automatic detection and assessment of dysarthria. Moreover, the duration feature showed variations due to gender, speaker, and phonetic context. Therefore, subsequent studies of the thesis focused on other state-of-art signal processing methods for the robust detection and assessment of dysarthria. The previous study anticipated that subtle variations in vocal tract system dynamics need to be captured by the feature representation for an accurate assessment of dysarthria. In this regard, narrow bandpass filtering based instantaneous spectral representation was investigated in this thesis to capture these variations. A new feature representation called perceptually enhanced narrow bandpass filtering based cepstral coefficient (PE-NBFCC) was proposed to detect and assess dysarthria. This study showed that the proposed features performed better than conventional features in terms of classification accuracy in both dysarthria detection and assessment tasks. Further to improve the performance of detection and assessment systems, phase information in the speech signals was explored. In this regard, the importance of analytic phase information was examined by proposing narrow bandpass filter bank-based instantaneous frequency cepstral coefficients (NBFB-IFCC). The proposed features showed better performance compared to standard features in terms of accuracy and F1-Score. The score-level fusion of proposed features with the magnitude spectral features improved the classification accuracy, which indicates the complementary information present in the analytic phase feature. Finally, this thesis explored the excitation features extracted using epoch-based speech processing. This study proposed a method for epoch extraction, namely zero-phase pitch selective bandpass filtering (ZP-PSBF). This study showed that the importance of the excitation source features for the detection and intelligibility assessment. The issues addressed in this thesis are summarized as follows: • Acoustic analysis of rhotic approximants was made, and the relation betwee

Abstract

In this thesis, we address the novel and highly challenging problem of estimating the layout of a complex urban driving scenario. Given a single color image captured from a driving platform, we aim to predict the bird’s eye view layout of the road, lanes, sidewalks and other traffic participants. The estimated layout should reason beyond what is visible in the image, and compensate for the loss of 3D information due to the perspective projection. We dub this problem amodal scene layout estimation, which involves hallucinating scene layout for even parts of the world that are occluded in the image. Firstly, we present MonoLayout, a deep neural network for real-time amodal scene layout estimation from a single image.MonoLayout maps a color image of a scene into a multi-channel occupancy grid in bird’s eye view, where each channel represents occupancy probabilities of various scene components. We represent scene layout as a multi-channel semantic occupancy grid, and leverage adversarial feature learning to “hallucinate” plausible completions for occluded image parts. We extend several state-of- the-art approaches for road-layout estimation and vehicle occupancy estimation in bird’s eye view to the amodal setup and thoroughly evaluate against them. By leveraging temporal sensor fusion to generate training labels, we significantly outperform current art over a number of datasets. Next, we aim to provide a comprehensive benchmark for the task of amodal layout estimation on KITTI and Argoverse datasets and also shift our attention towards estimating more fine-grained atr- ributes such as lanes, crosswalks, vehicles, etc. . To this end, we introduce AutoLay, a new dataset for amodal layout estimation in bird’s eye view. AutoLay includes precise annotations for (amodal) layouts for 44 sequences from the KITTI dataset. In addition to fine-grained attributes such as lanes, sidewalks, and vehicles, we also provide detailed semantic annotations for 3D pointclouds. To foster reproducibil- ity and further research in this nascent area, we open-source implementations for several baselines and current art. Further, we propose VideoLayout, a real-time neural net architecture that leverages tempo- ral information from monocular video, to produce more accurate and consistent layouts. VideoLayout achieves state-of-the-art performance on AutoLay benchmark, while running in real-time. We show the potency of the aformentioned methods using several abltaion studies and also show applications of amodal layout estimation on downstream tasks.

Abstract

In this thesis, I investigate the effects of certain psychoactive substances on the brain and propose a novel method to deal with addiction. The first section deals with the effect of marijuana on the different hippocampus subfields. The hippocampus has the highest concentration of cannabinoid receptors, the primary binding site for THC, the primary pscyhoactive constituent of marijuana. But this concentration varies across the subfields. We also check for the effects of varied consumption in this study by splitting the cannabis consuming group into two subgroups: heavy consumers and light consumers. The next section deals with a method to administer transcranial magnetic stimulation in patients with addiction and depression disorders amongst others. We focus on using a personalised locus rather than a population driven locus. The main issue faced is the highly variable nature of the dorsolateral prefrontal cortex, the target site for the administration. Subsequently, we highlight the importance of preprocessing and quality control in the processing of magnetic resonance images in both humans and non-human primates. Existing approaches in this area fail to deal with multiple artifacts often encountered while working on these scans. We then use quality control to develop a novel pipeline to preprocess macaque structural images. We also use quality control in the development of a state of the art pipeline for preprocessing and processing human diffusion weighted tensor images.

Abstract

The human body is a complex structure consisting of multiple organs, muscles and bones. However, in regard to modeling human motion, the information about fixed set of joint locations (with fixed bone length constraint) is sufficient to express the temporal evolution of poses. Thus, one can represent a human motion/activity as temporally evolving skeleton sequence. In this thesis, we primarily explore two key problems related to modeling of human motion, the first is efficient indexing & retrieval of human motion sequences and the second one is automated generation/synthesis of novel human motion sequences given input class prior. 3D Human Motion Indexing & Retrieval is an interesting problem due to the rise of several data-driven applications aimed at analyzing and/or re-utilizing 3D human skeletal data, such as data-driven animation, analysis of sports biomechanics, human surveillance, etc. Spatio-temporal articulations of humans, noisy/missing data, different speeds of the same motion, etc. make it challenging and several of the existing states of the art methods use hand-craft features along with optimization-based or histogrambased comparison in order to perform retrieval. Further, they demonstrate it only for very small datasets and few classes. We make a case for using a learned representation that should recognize the motion as well as enforce a discriminative ranking. To that end, we propose, a 3D human motion descriptor learned using a deep network. Our learned embedding is generalizable and applicable to real-world data - addressing the aforementioned challenges and further enables sub-motion searching in its embedding space using another network. Our model exploits the inter-class similarity using trajectory cues and performs far superior in a self-supervised setting. State of the art results on all these fronts is shown on two large scale 3D human motion datasets - NTU RGB+D and HDM05. In regard to the second research problem of Human Motion Generation/Synthesis, we aimed at long-term human motion synthesis that can aid to human-centric video generation [9] with potential applications in Augmented Reality, 3D character animations, pedestrian trajectory prediction, etc. Longterm human motion synthesis is a challenging task due to multiple factors like long-term temporal dependencies among poses, cyclic repetition across poses, bi-directional and multi-scale dependencies among poses, variable speed of actions, and a large as well as partially overlapping space of temporal pose variations across multiple class/types of human activities. This paper aims to address these challenges to synthesize a long-term (> 6000 ms) human motion trajectory across a large variety of human activity 1Video for Human Motion Synthesis. 2Video for Human Motion Indexing and Retrieval. classes (> 50). We propose a two-stage motion synthesis method to achieve this goal, where the first stage deals with learning the long-term global pose dependencies in activity sequences by learning to synthesize a sparse motion trajectory while the second stage addresses the synthesis of dense motion trajectories taking the output of the first stage. We demonstrate the superiority of the proposed method over SOTA methods using various quantitative evaluation metrics on publicly available datasets. In summary, this thesis successfully addressed two key research problems, namely, efficient human motion indexing & retrieval and long-term human motion synthesis. In doing so, we explore different machine learning techniques that analyze the human motion in a sequence of poses (or called as frames), generate human motion, detect human pose from an RGB image, and index human motion.

Abstract

Recently, a lot of the standard algorithms in machine learning and computer vision need to be redesigned for handling large scale problems. In this thesis, we explore one such fundamental algorithm, which is inherent in almost every multi-view reconstruction system pipeline in computer vision. Recently, bundle adjustment has been the key component in Structure-from-Motion (SfM) problem. Bundle adjustment (BA) jointly optimizes the camera parameters and feature points parameters according to an objective function of reprojection errors. This joint optimization of camera parameters and feature points paramters for multi-view reconstruction formulate the BA problem as a non-linear least squares problem which is solved by some variant of the traditional Levenberg-Marquardt (LM) algorithm [44]. Most of the computation in LM goes into repeatedly solving the normal equations that arise as a result of linearizing the objective function. Some widely used methods like Cholesky Decomposition and Preconditioned Conjugate Gradients [25], to solve these normal equations still face some challenges in robustness, accuracy, and efficiency. In this thesis, we propose a deflated algebraic two-grid method has been used as a preconditioner for Generalized Minimal Residual Method (GMRES) [58], for solving these normal equations arising in the Bundle Adjustment process. We show that the proposed method is several times faster than iterative schur and jacobi schur [2] which are considered as state-of-the-art methods to solve the Bundle Adjustment problem. We verify the competence of our algorithm by benchmarking it against the stateof-the-art methods on publicly available Bundle Adjustment in the Large (BAL) dataset [3].

Abstract

As demands for high volume data exchanges increase in many memory applications, significant efforts have been devoted to improving I/O interface performance to achieve high bandwidth. However, the impedance mismatch between I/O driver and a transmitting channel causes signal reflection, interfering with the incoming data in terms of overshoot, undershoot, or ringing. I/O interface requires a minimum variation of output I/O driver impedance over process, temperature, and voltage variations to ensure high-quality signal integrity. The traditional approach of using an external resistor component to generate a constant reference for impedance calibration is not an effective solution considering the constraints due to shrinking package form factor and low-cost limitations of memory applications. This thesis proposes a novel I/O driver impedance calibration architecture and its circuit design using internally generated constant current reference. Also, designed a current mode Bandgap Reference circuit, which generates a constant current across Process and Temperatures variations. A voltage detect circuit is proposed to capture Supply Voltage variations and to compensate the current reference accordingly. To minimize the error in calibration codes, a low offset comparator is designed. IO driver with a combination of MOS device and a series poly resistor is used for better linearity in impedance. This thesis proposes and implements an independent calibration of Pullup and Pulldown IO drivers. The pullup driver calibration codes are not reused for pulldown driver calibration. It reduces the mismatch in impedance of Pullup and Pulldown drivers by eliminating propagation of any quantization error in Pullup calibration codes to Pulldown driver calibration. The proposed calibration scheme and the whole circuit was designed and implemented in 28nm standard CMOS technology using 1.8V supply voltage. In spice simulations of the proposed calibration design/scheme, the impedance variation of both pullup and pulldown driver is approximately +/-7% across process, temperature, and voltage variations. This reduced IO driver impedance variation is achieved using the proposed internally generated constant current source without using any external component resistor.

Abstract

Autonomous driving has evolved with the advent of deep learning and increased computational capacity in the past two decades. While many tasks have evolved that help autonomous navigation, understanding on-road vehicle behaviour from a continuous sequence of sensor data (camera or radar) is an important and useful task. Determining state of other vehicles and their motion helps in decision making for the ego-vehicle (Self/Our Vehicle). Such systems can be a part of driver assistance systems or even path planners for vehicles to avoid obstacles. Instances of such decision making could be, deciding to apply brakes incase an external agent is moving into our lane or coming opposite to us. Another instance would be keeping distance from an aggressive driver who is changing lanes and overtaking frequently could be helpful and safer. Specifically, our proposed methods classify the behaviours of on-road vehicles into one of the following categories-{Moving Away, Moving Towards Us, Parked/stationary, Lane Change, Overtake}. Many current methods leverage 3D depth information using expensive equipment, such as Lidar and complex algorithms. In this thesis, we propose a simpler pipeline for understanding vehicle behaviour from a monocular (single camera) image sequence or video. This will help in easier installation of the equipment and helps in Driver Assistance systems. A simple video along with Camera parameters, scene semantics (Detected Objects in the images) are used to get information about the objects of interest (vehicles) and other static objects like lanes/poles in the scene. We consider detecting objects across frames as the pre-processing pipe-line and propose two main methods, 1.Spatio-temporal MRGCN (Chapter:3) and 2.Relational-Attentive GCN (Chapter:4) for behaviour detection of vehicles. We divide our process of identifying behaviours into learning positional information of vehicles first, and then observing them across time to make a prediction. The positional information is encoded by a Multi-Relational Graph Convolutional Network (MR-GCN) in both the methods. Temporal information is encoded by recurrent networks in Method 1, while it is formulated inside the graph in Method 2. We also showcase how Attention is an important aspect in both our methods through our experiments. The proposed frameworks can classify a variety of vehicle behaviours to high fidelity on datasets that are diverse and include European, Chinese and Indian on-road scenes. The framework also provides for seamless transfer of models across datasets without entailing re-annotation, retraining and even fine-tuning. We provide comparative performance gain over baselines and detail a variety of ablations to showcase the efficacy of the framework.

Abstract

Autonomous cars and fixed-wing aerial vehicles (FWV) have the so-called non-holonomic kinematics which non-linearly maps control input to states. As a result, trajectory optimization using such a motion model becomes highly non-linear and non-convex. In this work, we present two alternate methods to efficiently solve this trajectory optimization problem. The first approach is based on the path velocity decomposition paradigm. The optimization underlying this approach has a two layer structure wherein first, an appropriate path is computed for the vehicle followed by the computation of optimal forward velocity along it. A key feature of this approach is that it offloads most of the responsibility of collision avoidance to the velocity optimization layer for which computationally efficient formulations can be derived. In particular, we extend our previously developed concept of time scaled collision cone (TSCC) constraints and formulate the forward velocity optimization layer as a convex quadratic programming problem. In the second approach, we improve the computational tractability of the problem by reformulating trajectory optimization in terms of a set of bi-convex cost and constraint functions along with a non-linear penalty. The bi-convex part acts as a relaxation for the non-holonomic trajectory optimization while the residual of the penalty dictates how well its output obeys the non-holonomic behavior. We adopt an alternating minimization approach for solving the reformulated problem and show that it naturally leads to the replacement of the challenging non-linear penalty with a globally valid convex surrogate. Along with the common cost functions modeling goal-reaching, trajectory smoothness, etc., the proposed optimizer can also accommodate a class of non-linear costs for modeling goal-sets, while retaining the bi-convex structure. We perform validation of first approach on autonomous driving scenarios to compute lane change, overtaking and merging maneuvers among multiple dynamic obstacles. We later benchmark the second approach against those based on sequential quadratic programming (SQP) and interior-point methods on common examples from autonomous driving and FWV navigation and show that it produces solutions with similar cost as the former while significantly outperforming the latter. Furthermore, as compared to both SQP and interior-point optimizers, it achieves more than 20x reduction in computation time.

Abstract

In recent years there has been an increase in the generation of health care documents such as clinical trials, discharge summaries, and Electronic Health Records (EHRs). These documents contain a lot of actionable data buried in them. Actionable data includes a set of events and activities that occur in health care processes. This valuable information has led to an increased scope for research on biomedical literature. However, most of the data reside in the form of free text which makes it difficult to extract useful information. The thesis develops methods to automatically extract semantics from the health care documents in an effort to check conformance of the treatment processes with standard treatment guidelines. Discharge summary is one of the major sources of information about the treatment process. A discharge summary contains information about a patient’s one or more encounters with health care service providers, stored electronically to share across different stakeholders in the health care system. The discharge summary has summarized information that includes a wide range of information like chief complaint, physical examination, vital signs, lab test results, recommended medications, and discharge status. Text analytics over these documents has various applications and would help the caregivers to provide better health care. The main theme of the thesis is to automatically extract medical semantics from discharge summaries. We focus on semantics such as medical entities, attributes of medical entities and relationships between these entities. Further, we illustrate an application of these semantics on a Treatment Process Conformance Checking usecase. Automatically identifying medical entities from biomedical literature is referred to as Biomedical Named Entity Recognition (BMNER). BMNER is one of the important tasks in the field of biomedical text mining. BMNER involves identifying continuous named entities and discontinuous named entities. Discontinuous entities are comprised of two or more non-consecutive components. For example, in a sentence, “The aortic root is moderately dilated.”, the disjoint entity “aortic root...dilated” is composed of two components, “aortic root” and “dilated”. Most of the prior works on BMNER were based on feature-dependent machine learning techniques and focused only on continuous named entities and not discontinuous entities. Traditional BIO tagging schema is unable to tag sentences with discontinuous entities. In this thesis, we propose a novel systematic BIODT tagging schema to identify both continuous and discontinuous named entities. We explore deep learning models that require limited feature engineering for tagging the entities. Our results illustrate that our BIODT tagging schema performs better than traditional BIO and other tagging schemas and overcomes label sparsity problem in identifying both continuous and discontinuous biomedical entities. We also show that our neural network model with BIODT tagging schema has shown superior performance than state-of-the-art methods on CLEF 2013 and SemEval 2013 datasets which were based on feature-dependent machine learning techniques. Mere medical entities cannot give enough information for understanding the condition of the patient. In a given context, characteristic of a medical entity is based on different attributes like temporal information, severity and progression of the disease. In this work, we consider ten attributes that allow us to understand the main details regarding the condition of the patient. They are Negation Indicator, Subject Class, Uncertainty Indicator, Course Class, Severity Class, Conditional Class, Generic Class, Body Location, DocTime Class, and Temporal Expression. In this thesis, we present a methodology with rule-based and machine learning approaches to identify each of these attributes. We evaluate our methodology on ShARe/CLEF eHealth Evaluation Lab 2014 Challenge dataset on attribute level and system level accuracy. Mining relationships between treatment(s), test(s) and medical problem(s) is vital in the biomedical domain. This helps in various applications such as decision support system, safety surveillance, and new treatment discovery. In this thesis, we propose a deep learning approach that utilizes both word level and sentence-level representations to extract the relationships between treatment and problem. While deep learning techniques demand a large amount of data for training, we make use of a rule-based system particularly for relationship classes with fewer samples. Our final relations are derived by jointly combining the results from deep learning and rule-based models. Our system achieved a promising performance on the relationship classes of I2b2 2010 relation extraction task. Finally, we employ the above pipeline of tasks such as Biomedical Named Entity Recognition, Medical Entity Attribute extractio