Abstract

Majority of speech and natural language processing applications rely on word and document representations (or embeddings). The document embeddings encode semantic information which makes them suitable for tasks such as topic identification (document classification), topic discovery (document clustering), language model adaptation, and query-based document retrieval. These embeddings are usually learned from widely available un-labelled data; hence generative or probabilistic topic models which aim to capture the distribution of data are suitable. Although there exist several probabilistic and neural network-based topic models to learn these embeddings, they often ignore to capture the uncertainty in the estimated embeddings. Thus, any error in the estimation of these embeddings affects the performance in downstream tasks. The uncertainty in the embeddings is usually due to shorter, ambiguous or noisy sentences/documents. This thesis presents model(s) for learning to represent document embeddings in the form of Gaussian distributions, thereby encoding the uncertainty in their covariances. Further, these learned uncertainties in embeddings are exploited by the proposed generative Gaussian linear classifier for topic identification. This thesis proposes to use subspace multinomial model (SMM), a simple log-linear model for learning document embeddings. Experiments on 20Newsgroups text corpus show that the embeddings extracted from SMM are superior when compared to popular topic models such as latent Dirichlet allocation, sparse topical coding in topic identification and document clustering tasks. Using the variational Bayes framework on SMM, the model is able to infer the uncertainty in document embeddings, represented by (posterior) Gaussian distributions. Additionally, the common problem of intractability which appears while performing variational inference in mixed-logit models is addressed using Monte Carlo sampling via the re-parametrization trick. The resulting Bayesian SMM achieves state-of-the-art perplexity results on 20Newsgroups text and Fisher speech corpora. The proposed generative classifier exploits the learned uncertainty in the document embeddings; and achieves state-of-the-art classification results on the aforementioned corpora as compared to other unsupervised topic and document models. Furthermore, this thesis presents a multilingual extension of the Bayesian SMM for zero-shot cross-lingual topic identification. The proposed model achieves superior classification results when compared to the systems based on multilingual word embeddings and neural machine translation inspired sequence-to-sequence bidirectional long-short term memory models.

Abstract

With the increase in number and size of the problems in the computer vision domain, a lot of research has focused on speeding up the algorithms in these domains. Most of these algorithms require the solution of a linear system at some point. This thesis explores the application of numerical linear algebra(NLA) techniques for increasing the efficiently of algorithms related to the bundle adjustment[62] problem. Since, the dimensions of the linear systems associated with these problems is large, iterative methods such as conjugate gradients(CG) and generalized minimal residual(GMRES), are used. However, the efficiency of these methods depends on the condition number(defined as the ratio of the largest eigenvalue to the smallest eigenvalue) of the coefficient matrix. In general, the matrices associated with these problems are very badly conditioned. This motivates the use of preconditioning matrices, or preconditioners, which are close to the original matrix in some sense, and simplifies the solution of the linear systems. In this thesis, some novel preconditioners are developed and their effects on the efficiency of the algorithms are presented.Two NLA techniques, namely the domain decomposition method and the multigrid method, are used to develop preconditioners for the bundle adjustment problem. This is the first time these techniques have been applied to this problem. Another method is explored in this thesis that is based on deflation of unfavourable eigenvectors. These preconditioners are integrated with popular solvers like SSBA1 and Ceres[2] and their improvements are demonstrated experimentally

Abstract

The internet has caused a revolutionary information explosion and we have more resources available than ever before. The size of just the English language Wikipedia articles is about 159.69 GB. As the resources we consume keep getting bigger and more complex, we need systems to query these datasets. Initially, to query large databases, simple Information Retrieval techniques like fetching keywords and combining them with Boolean logic results were used. But now, we have search engines that can take queries in the form of multiple keywords and even questions. The retrieval side of these search engines also started with simple Document Retrieval methods like document frequencies and inverse document frequencies and has evolved to complex ranking methods based on neural networks that can fetch entire questions. Question Answering is a major field in NLP. It combines query formulation and document retrieval and in some cases, summarization of the retrieved documents. We now have complete end-to-end based systems that can fetch a suitable answer from a corpus. These systems can be trained on large datasets - both closed domain or open domain. However, these systems rely heavily on curated datasets which require a lot of manual labour and resources. For Indian languages where reliable and substantial datasets are scarce, neural networks that rely purely on data cannot be used. Hence, providing more data and introducing data-independent techniques becomes important. We shall explore both these aspects. First, we present a Reading Comprehension Task in Hindi - HindiRC. The dataset has been divided into different grades based on reading proficiency and we perform the baseline experiment on each grade separately which shows the increasing level of difficulty. Using various linguistic cues and metrics we further prove that the grades are reflective of linguistic complexity. In addition to addressing the data scarcity, we propose a Hindi Question Generation methodology. The rule-based method is based on Computational Paninian Grammar Framework relations. No additional resources are required and it can be used to increase the number of questions for Hindi datasets. We also prove that the generation method tends to overgenerate questions; further inflating the number. Along with the dataset and question generation method, we aim to provide more resources for Question Answering in Hindi.

Abstract

Real-world images often suffer from variation in weather conditions such as rain, fog, snow, and temperature. These variations in the atmosphere adversely affect the performance of computer vision models in real-world scenarios. This problem can be bypassed by collecting and annotating images for each weather. However, collecting and annotating images in such conditions is an extremely tedious task, which is time-consuming as well as expensive. So, in this work, we address the forementioned problems. Among all the weather conditions, we focus on the distortions in the image caused by high temperature, also known as atmospheric turbulence. These distortions introduce geometrical deformation around the boundaries of an object in an image which causes a vision algorithm to perform poorly and pose a major concern. Hence, in this thesis, we address the problem of artificially generating atmospheric turbulence and restoring the images from it. In the first part of our work, we attempt to model atmospheric turbulence. Since such models are critical to extending computer vision solutions developed in the laboratory to real-world use cases. And, simulating atmospheric turbulence by using statistical models or by computer graphics is often computationally expensive. To overcome this problem, we train a generative adversarial network(GAN) which outputs an atmospheric turbulent image by utilizing less computational resources than traditional methods. We propose a novel loss function to efficiently learn the atmospheric turbulence at the finer level. Experiments show that by using the proposed loss function, our network outperforms the existing state-of-the-art image to image translation network in turbulent image generation. In the second part of the thesis, we address the ill-posed problem of restoring images degraded due to atmospheric turbulence. We propose a deep adversarial network to recover the images which are distorted due to atmospheric turbulence and show the applicability of restored images in several tasks. Unlike previous methods, our approach neither uses any prior knowledge about atmospheric turbulence conditions at inference time nor requires the fusion of multiple images to get a single restored image. To train our models, we synthesized turbulent images by following a series of efficient 2D operations. Thereafter, using our trained models we run inference on real and synthesized turbulent images. Our final restoration models DT-GAN+ and DTD-GAN+ qualitatively and quantitatively outperforms the general state-of-the-art image-to-image translation models. The improved performance of our model is due to the use of optimized residual structures along with channel attention and sub-pixel mechanism which exploits the information between the channels and removes atmospheric turbulence at the finer level. We also perform extensive experiments on restored images by utilizing them for downstream tasks such as classification, pose estimation, semantic keypoint estimation, and depth estimation. In the third part of our work, we study the problem of the semantic segmentation model in adapting to hot climate cities. This issue can be circumvented by collecting and annotating images in such weather conditions and training segmentation models on those images. But, the task of semantically annotating images for every environment is painstaking and expensive. Hence, we propose a framework that improves the performance of semantic segmentation models without explicitly creating an annotated dataset for such adverse weather variations. Our framework consists of two parts, a restoration network to remove the geometrical distortions caused by hot weather and an adaptive segmentation network that is trained on an additional loss to adapt to the statistics of the ground-truth segmentation map. We train our framework on the Cityscapes dataset, which showed a total IoU gain of 12.707 over standard segmentation models. In the last part of our work, we improve the performance of our joint restoration and segmentation network via a feedback mechanism. In, the previous approach the restoration network does not learn directly from the errors of the segmentation network. In other words, the restoration network is not task aware. Hence, we propose a semantic feedback learning approach, which improves the task of semantic segmentation giving a feedback response into the restoration network. This response works as an attend and fix mechanism by focusing on those areas of an image where restoration needs improvement. Also, we proposed loss functions: Iterative Focal Loss (iFL) and Class-Balanced Iterative Focal Loss (CBiFL), which are specifically designed to improve the performance of the feedback network. These losses focus more on those samples that are continuously miss-classified over successive iterations. Our approach gives a gain of 17.41 mIoU over the standard segmentation model, in

Abstract

Advancements in 3D sensing and reconstruction has made a huge leap for modeling large-scale environments from monocular images using structure from motion (SfM) and simultaneous localization and mapping (SLAM) algorithms. SfM and SLAM based 3D reconstruction has applications for digital archival and modeling of real-world objects and environments, visual localization for geo-tagging and information retrieval, and mapping and navigation for robotic and autonomous driving applications. In this thesis, we address problems in the area of large-scale structure from motion (SfM) for 3D reconstruction and localization. We introduce new methods for improving efficiency and accuracy of state-of-the-art pipeline for structure from motion. Large-scale SfM pipeline deals with large unorganized collections of images pertaining to a particular geographical site. These image collections are formed by either retrieving relevant images using textual queries from the Internet, or can be captured for the specific purpose of 3D modeling, mapping, and navigation. Internet image collections tend to be more noisy and present more challenges for reconstruction as compared to datasets captured with specific intention to reconstruct. In this thesis, we propose methods that help with organizing these large, unstructured, and noisy images into a structure that is useful for SfM methods, a match-graph (or a view-graph). We first propose a geometry-aware two stage approach for pairwise image matching that is both more efficient and superior in quality of correspondences. We then extend this idea to SfM pipeline and present an iterative multistage framework for coarse to fine 3D reconstruction. Finally, we suggest that a key to solving many of the reconstruction problems is to address the problem of filtering and improving the view-graph in a way that is specific to the underlying problem. To this effect, we propose a unified framework for view-graph selection and show its application to achieve multiple reconstruction objectives.

Abstract

Natural Answer Generation(NAG) is the task of generating full-length sentential answers to factbased questions. This task is a desired attribute of conversational and task-oriented dialogue systems which aim to interact with the user in natural responses. Moreover, multi-modal interfaces enabling speech based input in addition to providing natural responses enhance the ease with which the user interacts with such systems. NAG systems have been widely studied in knowledge-base (KB) question answering tasks where the system generates natural answers to fact-based questions utilizing the structured information in the KB. Even though the task of generating natural answers to textual questions has been explored in recent years, there is little research on natural sentence generation over spoken content. To address this challenge, we propose the task of full-length natural answer generation from spoken question. We approach the task iteratively with sequential sub-tasks. First we study a pointergenerator based full-length natural answer generator system which generates natural answer from a textual question and factoid answer. The pointer-generator system effectively copies facts from the input addressing the challenges of out-of-vocabulary words and named-entities present in datasets. To train and evaluate our system, we develop a dataset of 315,000 samples comprising of question, factoid answer and full-length answer triples to train the system. We test our system over cross-domain samples extracted from a knowledge-base dataset such as Freebase and a machine comprehension dataset such as NewsQA. The system achieves a BLEU score of 74.05 and a Rogue-L score of 86.25. As a second sub-task, we study the effect of spoken input on dialogue systems. Spoken dialogue systems typically use one or several (top-N) ASR sequence(s) for inferring the semantic meaning and tracking the state of the dialogue, either with an end-to-end system or with a Spoken Language Understanding (SLU) module associated to a dialogue state tracker (DST). However ASR graphs, such as confusion networks (confnets), provide a compact representation of a richer hypothesis space than a top-N ASR list. In this sub-task, we study the benefits of using confusion networks with a state-of-the-art neural dialogue state tracker (DST). We encode the 2-dimensional confnet into a 1-dimensional sequence of embeddings using an attentional confusion network encoder which can be used with any DST system. Our confnet encoder is plugged into the state-of-the-art ’Global-locally Self-Attentive Dialogue State Tacker’ (GLAD) model for DST and obtains significant improvements in both accuracy and inference time compared to using top-N ASR hypotheses. Subsequently, we address the next sub-task of generating full length natural answer from spoken question. To the best of our knowledge, this is the first attempt towards generating full-length natural answers from spoken content. We approach this task by representing the spoken sequence compactly as a confusion network extracted from a pre-trained Automatic Speech Recognizer. We develop a pointer-generator system over the confusion network using the pre-assigned ASR scores and global attention to copy words from the confusion network and textual factoid answer. We release a large-scale dataset of 259,788 samples of spoken questions, their factoid answers and corresponding full-length textual answers called the ConfNet2Seq dataset. Following our proposed approach, we achieve comparable performance with best ASR hypothesis. Using ASR scores to copy words from the confusion network has the limitation that only the best ASR hypothesis is always chosen to be copied from the confusion network. The final sub-task addresses this issue with a novel hierarchical pointer-generator network which copies words from alternative hypothesis of the confusion network instead of being limited to the best ASR hypothesis. We analyse the benefits of using an ASR graph over the hierarchical pointer-generator network for the task of full-length natural answer generation from spoken questions. We experiment over increasingly noisy data to evaluate the performance of our system in noisy setting and show that our system outperforms the ConfNet2Seq system on the ConfNet2Seq dataset and performs similar to the ConfNet2Seq system over very noisy data. We also perform cross-dataset evaluation to evaluate the efficacy of our method over new domains.

Abstract

The internet has heralded an age of information explosion. A simple search query yields thousands of results. Each of these, in turn, is linked to a lengthy and verbose document. There is an immediate need to find faster and efficient ways to consume this ever-growing ocean of information. The purpose of summarization is to generate a shorter representation of the input that captures its gist and preserves its intent. In this work, we explore extractive summarization techniques, which involve identifying and extracting the most informative parts of a text. Most extractive summarization systems have been developed for the domains of newswire articles and scientific texts. These systems usually operate by ranking all the source sentences, according to some heuristic or metric, and then select the top sentences as the summary. They work well in domains where the discourse revolves around a central theme and information is often enforced by reiteration across several sentences. However, documents such as fictional narratives do not talk about a single topic. They describe a sequence of events and often contain dialogue. They do not contain repetitive information and each sentence contributes to developing the plot further. Hence, selecting a subset of sentences does not accurately capture the story. In the first part of this thesis, we discuss telegraphic summarization. Just like a telegram, a telegraphic summary does not contain complete sentences. Instead, shorter phrases are extracted across multiple sentences to capture the crux of the document. This summarization technique is better suited for the domain of fiction. We describe a set of guidelines to create such summaries. These guidelines were used to annotate a gold corpus of 200 English short stories. In the later part of this thesis, we model the task of telegraphic summarization as the well-known sentence compression task. Sentence compression is the task of shortening a sentence while retaining its meaning. Most methods proposed for this task rely on labeled or paired corpora (containing pairs of verbose and compressed sentences), which are often expensive to collect. To overcome this limitation, we present a novel unsupervised deep learning framework (SCAR) for deletion-based sentence compression. SCAR is primarily composed of two encoder-decoder pairs: a compressor and a reconstructor. The compressor generates a sequence of zeroes and ones, representing token deletion decisions, used to mask the input sentence. The reconstructor tries to regenerate the input from the masked sentence. The model is trained completely on unlabeled data and does not require additional inputs such as explicit syntactic information or optimal compression length. SCAR’s merit lies in the novel Linkage loss function, which correlates the compressor and its effect on reconstruction, guiding it to drop inferable tokens. SCAR achieves higher ROUGE scores than the existing state-ofthe-art methods and baselines on benchmark datasets. We also conduct a user study to demonstrate the application of our model as a text highlighting system. Using our model to underscore salient information facilitates speed-reading and reduces the time required to skim a document.

Abstract

With the exponential increase in amount of data being stored in the cloud, a lot of distributed storage systems have started moving to erasure coding based storage schemes due to their advantage of providing the same reliability as replication with a smaller storage overhead. Locally recoverable codes have started being the defacto code of choice for their ability to correct a small number of erasures by only accessing a small number of other coordinates. Maximally recoverable codes are a class of codes which recover from all potentially recoverable erasure patterns given the locality constraints of the code. In earlier works, these codes have been studied in the context of codes with locality. The notion of locality has been extended to hierarchical locality, which allows for locality to gradually increase in levels with the increase in the number of erasures. We consider the locality constraints imposed by codes with two-level hierarchical locality and define maximally recoverable codes with data-local and local hierarchical locality. We derive certain properties related to their punctured codes and minimum distance. We give a procedure to construct hierarchical data-local MRCs from hierarchical local MRCs. We then provide a construction of hierarchical local MRCs for all parameters. We then provide an alternate construction using tensor product codes that construct the code in a smaller field. This construction works for a restricted set of parameters. Research shows that the most commonly deployed LRCs have low number of global parities. Hence, we then consider three cases, with a small (and sometimes fixed) number of global and mid-level parities and provide a separate construction for them. These constructions use smaller fields still.

Abstract

This thesis proposes a novel pose-graph based formulation to map moving object trajectories onto a stationary global reference frame where the moving objects are observed from a single moving camera. This is performed in a multi-body fashion as a consequence of which we obtain accurate estimations to both the ego-camera trajectories as well as trajectories to multiple dynamic participants in the scene. This set up is dovetailed to on-road dynamic vehicles observed from a monocular sensor mounted on a moving ego-car such as in the KITTI dataset. The original triangulation problem is intractable in this scenario as it is impossible to triangulate a moving object from a single moving camera unambiguously unless there are appropriate restrictions on the object motion. The problem of unobservability also manifests in the form of the relative scale problem in classical multi-body SFM/SLAM formulations wherein there exists an unresolved scale factor (called as relative scale) such that a family of infinitely many solutions exist for any pair of motions in the scene. This prevents accurately representing the moving vehicle and camera in the same stationary frame. We overcome this relative scale factor by leveraging single-view metrology, advances in deep learning, and category-level shape estimation. We show in this paper by invoking single view reconstruction techniques, that moving objects can be represented in the same global frame in which the camera trajectory is represented without any assumptions or restrictions on object motion. More specifically we solve for the relative scale problem through a factor graph formulation where the nodes include camera and moving object poses and thereby obtain trajectories of the moving object in the starting camera frame. We use Gaussian Process based motion model prediction and lane constraints to further improve the trajectory estimates and show performance gain with previous formulations that have attacked this problem. This optimization helps us reduce the average error in trajectories of multiple bodies over real-world datasets, such as KITTI. To the best of our knowledge, our method is the first practical monocular multi-body SLAM system to perform dynamic multi-object and ego localization in a unified framework in metric scale

Abstract

Graph algorithms are one of the most robust and powerful algorithms to solve many contemporary real-world problems. The humongous real-world graph networks should be parallelized to study and gain understanding from them. Literature consists of several studies that proposed multiple methods to compute various centrality measures for a multi-core system. In this thesis, we study about farness and closeness centrality measures. To the best of our knowledge, this is the first work in parallel algorithms with respect to farness-centrality. We utilize the information about the structure of the graph to abstain from doing repetitive tasks. In this thesis, we propose a parallel algorithm to compute the farness-centrality value using graph optimizations while avoiding the redundancy on the graph using the properties detailed in this thesis below. It is inevitably tough and time-consuming to calculate the values for each and every node in social graphs since they are too big to handle even after optimizations. So, we estimate the values by taking a trade-off on time vs. quality. We accept a little reduction in the quality of results prioritizing on time. We analyze our algorithm and reduction techniques on various classes of real-world graphs. However, as these graphs evolve and change, it is crucial to study how to update farness and closeness measures on changes to the underlying graph. In this thesis, we also present approaches to compute closeness-centrality on dynamic graphs. As existing work by Sarıyuce et al. (CLUSTER 2013) in this ¨ field is limited to a single edge update, we extend it to batch update. We implement our algorithms on Intel 24-core CPU. We conduct detailed experiments to study the impact of various parameters associated with our algorithms and their implementation. Our results indicate that on a collection of real-world graphs, our techniques improve the state-of-the-art with respect to updating closeness-centrality by a factor of 10 to 34 for a batch of 100 edges on dynamic graphs and 10 to 20 on static graphs over a corresponding adaptation of the work of Sarıyuce et al. (CLUSTER 2013) ¨ and MULTIBFS respectively.