Abstract

Text generation has shown tremendous promise recently, mainly attributed to the use of transformer architecture and the models pretrained on vast amounts of data. Multiple business scenarios today deploy neural-network-based models for natural language generation(NLG) tasks. However, this progress is limited to English and other high-resource(HR) languages, with NLG systems in low-resource(LR) languages far behind in terms of accuracy and fluency of generated text. This is due to several factors, such as lack of training data, lack of robust models supporting native script, and lack of linguistic resources as well. In this work, we extensively study an approach of cross-lingual NLG to tackle these challenges. Cross-lingual NLG implies exploiting the widely available data in HR language to generate the desired text in LR language. We focus on two significant tasks, i.e., fact-to-text and summarization, with a larger goal of generating Wikipedia article text in LR languages. We propose novel ways to build the datasets for the above tasks and also the approaches to generate text in LR languages. Firstly, we propose a novel task of cross-lingual fact-to-text generation(XF2T). Given the Wikidata facts in English, the system is expected to generate a sentence describing these facts in the desired language. To build a parallel dataset to train a model for the same, we explore several methods to link a fact from Wikidata to a sentence from Wikipedia, such as unsupervised, distantly supervised, and zero-shot learning-based approaches. We use the best approach to create the dataset XAlign of 0.55M instances across 12 Indian languages. Further, we implement transformer encoder-decoder and mT5 model as baselines using this dataset. In addition, we also explore the impact of task-specific pretraining, bilingual and monolingual models. We experiment with techniques to improve efficiency, such as structure-aware encoding of facts and fusing role-specific embeddings. We show that these approaches generate fluent and highly accurate sentences. Further, intending to generate longer text, we propose one more novel idea to generate the Wikipedia article section text using summarization. We leverage the citations available for each section on Wikipedia pages and build a parallel dataset for cross-lingual, multi-document, aspect-based summarization in 8 domains and 15 languages. In the first stage, i.e., extractive summarization, we aim to filter relevant sentences from a set of reference articles, for which we use saliency and graph-based methods. We experiment with recent SOTA models mT5 and mBART in the abstractive stage. Despite high noise in the input reference articles, we vi show that the system generates fluent and meaningful outputs. Although, in terms of content coverage and text coherency, models have a lot of scope for improvement. Overall, we work on various methods using cross-lingual NLG to advance the datasets and models in LR languages. We hope this work will boost more research in these critical areas in the future

Abstract

Language understanding has become crucial in different text classification tasks in Natural Language Processing (NLP) applications to get the desired output. Over the past decade, machine learning and deep learning algorithms have been evolving with efficient feature representations to give better results. The applications of NLP are becoming potent, domain, and language-specific. For resource-rich languages like English, the NLP applications give desired results due to the availability of large corpora, different kinds of annotated datasets, efficient feature representations, and tools. Due to the lack of large corpora and annotated datasets, many resource-poor Indian languages struggle to reap the benefits of deep feature representations. Moreover, adopting existing language models trained on large English corpora for Indian languages is often limited by data availability, rich morphological variation, syntax, and semantic differences. Most of the work being done in Indian languages is from a machine translation perspective. One solution is to use translation for re-creating datasets in low resource languages from English. But in case of Indian languages like Telugu, the meaning may change and some crucial information may be lost due to translation. This is because of their structural differences, morphological complexities, and semantic differences. In this thesis, our main objective is to mitigate the low-resource problem for Telugu. Overall, to accelerate NLP research in Telugu, we present several contributions: (1) A large Telugu raw corpus of 80,15,588 sentences (16,37,408 sentences from Telugu Wikipedia and 63,78,180 sentences crawled from different Telugu websites). (2) Annotated datasets in Telugu for sentiment analysis, emotion identification, hate speech detection, sarcasm identification, and clickbait detection. (3) For the Telugu corpus, we are the first to generate pre-trained distributed word and sentence embeddings such as Word2Vec-Te, GloVe-Te, FastText-Te, MetaEmbeddings-Te, Skip-Thought-Te. (8) We pre-trained different contextual language models for Telugu such as ELMo-Te, BERT-Te, RoBERTa-Te, ALBERT-Te, Electra-Te, and DistilBERT-Te, word and sentence embeddings using graph-based models: DeepWalk-Te and Node2Vec-Te, and Graph AutoEncoders (GAE). (4) We propose the multi-task learning model (MT-Text GCN) to reconstruct word-sentence graphs on TEL-NLP data while achieving multi-task text classification with learned graph embeddings. We show that our pre-trained embeddings are competitive or better than the existing multilingual pre-trained models: mBERT, XLM-R, and IndicBERT. Lastly, the fine-tuning of pre-trained models show higher performance than linear probing results on five NLP tasks. We also experiment with our pre-trained models on other NLP tasks available in Telugu (Named Entity Recognition, Article Genre Classification, Sentiment Analysis, and Summarization) and find that our Telugu pre-trained language models (BERT-Te and RoBERTa-Te) outperform the state-of-the-art system except for the sentiment task. We hope that the availability of the created resources for different NLP tasks will accelerate Telugu NLP research which has the potential to impact more than 85 million people. In this thesis, we aim at bridging the gap by creating resources for different NLP tasks in Telugu. These tasks can be extended to other Indian languages that are closer to Telugu culturally and linguistically by translating these resources without losing information like verb forms, cultural terms, vibhaktis etc. This is the first work that employs neural methods to Telugu language —a language that does not have good tools like NER, parsers and embeddings. Our work is the first attempt in this direction to provide good models in Telugu language by exploring different methods with available resources. It can also help the Telugu NLP community evaluate advances over more diverse tasks and applications. We open-source our corpus, five different annotated datasets (SA, EI, HS, SAR, and clickbait), lexicons, pre-trained embeddings, and code here1 . The pre-trained Transformer models for Telugu are available here2.

Abstract

Given a premise, the aim of Natural Language Inference is to identify a hypothesis as Entailed, Refuted, or Neutral. To do such classification, a model must acquire the ability to reason over the premise. While entailment tasks have been extensively studied with unstructured text as the premise, there is an increasing demand for learning to reason over semi-structured and organised data formats such as tables, knowledge graphs, databases, and combinations thereof. Structured data forms differ from unstructured text in the way they capture information and relationships – not just via language, but also through position and structure. Particularly, tables capture the connections between cells, which represent isolated distinct entities. Tabular data is organised so that items of the same kind are grouped together in rows, columns, or both. Consequently, it is straightforward to infer rankings, trends, unique items, and aggregate values from tabular data. These sorts of reasoning are specific to structured data formats, which makes inference on tables a difficult task requiring separate effort from inference on plain text. Creating challenging tabular inference data for supervision is necessary for mastering complex reasoning. Prior research in this sector has predominantly employed two data generating methodologies. The first technique is human annotation. This results in data that is inventive, fluent, and linguistically diverse. However, human annotation is costly and time-consuming, making it difficult to scale. The second form of data production is through synthetic means, where the data is generated using a defined set of rules or context-free grammar. This system is easily scalable in terms of both time and cost, but it lacks originality. Its results are predictable and adhere to predetermined patterns and fixed vocabulary. This research presents a framework for semi-automatically "recasting" existing tabular data in order to mitigate the drawbacks of both of the aforementioned data generation techniques. Existing data is perturbed, modified, and augmented through recasting to conform to the specifications of a given target task, which is Tabular Inference in this case. This framework is used to construct tabular NLI instances from five datasets that were originally designed for tasks such as table-to-text generation, tabular question answering, and tabular semantic parsing. To demonstrate the utility and quality of these datasets, this thesis explains how recasted data may be utilised as evaluation benchmarks and augmentation data to improve performance on tabular NLI tasks such as TabFact. In addition, this work evaluates the efficacy of models trained on recasted data in the zero-shot setting and examines performance trends across different types of recasted datasets. This thesis concludes with a discussion of the limitations and potential future paths of this field of study

Abstract

The reconstruction and analysis of 3D objects by computational systems has been an intensive and long-lasting research problem in the graphics and computer vision scientific communities. Traditional acquisition systems are largely restricted to studio environment setup which requires multiple synchronized and calibrated cameras. With the advent of active depth sensors like time-of-flight sensors, structured lighting sensors made 3D acquisition feasible. This advancement of technology has paved way to many research problems like 3D object localization, recognition, classification, reconstruction which demand innovating sophisticated/elegant solutions to match their ever growing applications. 3D human body reconstruction, in particular, has wider applications like virtual mirror, gait analysis, etc. Lately, with the advent of deep learning, 3D reconstruction from monocular images garnered significant interest among the research community as it can be applied to in-the-wild settings. Initially we started exploration of classification of 3D rigid objects due to availabilty of ShapeNet datasets. In this thesis, we propose an efficient characterization of 3D rigid objects which take local geometry features into consideration while constructing global features in the deep learning setup. We introduce learnable B-Spline surfaces in order to sense complex geometrical structures (large curvature variations). The locations of these surfaces are initialized over the voxel space and are learned during training phase leading to efficient classification performance. Later on, we primarily focus on rather challenging problem of non-rigid 3D human body reconstruction from monocular images. In this context, this thesis presents three principle approaches to address 3D reconstruction problem. Firstly, we propose a disentangled solution where we recover shape and texture of the 3D shape predicted using two different networks. We recover the volumetric shape of non-rigid human body shapes given a single view RGB image followed by orthographic texture view synthesis using the respective depth projection of the reconstructed (volumetric) shape and input RGB image. Secondly, we propose PeeledHuman - a novel shape representation of the human body that is robust to self-occlusions. PeeledHuman encodes the human body as a set of Peeled Depth and RGB maps in 2D, obtained by performing ray-tracing on the 3D body model and extending each ray beyond its first intersection. We learn these Peeled maps in an end-to-end generative adversarial fashion using our novel framework - PeelGAN. The PeelGAN enables us to predict shape and color of the 3D human in an end-to-end fashion at significantly low inference rates. Finally, we further improve PeelGAN by introducing a shape prior while reconstructing from monocular images. We propose a sparse and efficient fusion strategy to combine parametric body prior with a non-parametric PeeledHuman representation. The parametric body prior enforces geometrical consistency on the body shape and pose, while the non-parametric representation models loose clothing and handles self-occlusions as well. We also leverage the sparseness of the non-parametric representation for faster training of our network while using losses on 2D maps. We evaluate our proposed methods extensively on a number of datasets. In this thesis, we also introduce 3DHumans dataset, which is a 3D life-like dataset of human body scans with rich geometrical and textural details. We cover a wide variety of clothing styles ranging from loose robed clothing like saree to relatively tight-fitting shirt and trousers. The dataset consists of around 150 male and 50 unique female subjects. Total male scans are about 180 and female scans are around 70. In terms of regional diversity, for the first time, we capture body shape, appearance and clothing styles for the South-Asian population. This dataset will be released for research purposes.

Abstract

The Himalayan region, spanning 2,500 kilometers in northern India, is highly prone to seismic activity. Situated in seismic zones IV and V, this region experiences frequent and devastating earthquakes, responsible for 70% of the world's fatal landslides. Factors such as steep slopes, heavy rainfall, uneven topography, geological conditions, climate, and unplanned urbanization exacerbate the susceptibility of the Himalayan landscape to landslides during earthquakes. The ongoing collision between the Indian and Eurasian plates generates faults and stress, making the region a high-risk area for future earthquakes. Khattri's 1999 research suggests a 56% probability of a magnitude 8.5 or greater earthquake occurring in the Himalayan seismic gap within the next century. Therefore, earthquake- induced landslides are a significant concern, necessitating enhanced preparedness to mitigate social and economic setbacks. To assess slope stability under seismic conditions, researchers employ deterministic, probabilistic, and statistical techniques. While predicting earthquakes with absolute certainty is impossible, seismic hazard studies estimate expected ground motion levels. These studies play a vital role in identifying ground shaking intensities that can trigger slope failures, quantifying hazards associated with specific locations. Integrated seismic hazard assessments, considering slope properties, enable the evaluation of likely ground motion scenarios. Therefore, conducting comprehensive and up-to-date seismic hazard studies is crucial to assess future landslide hazards in the earthquake-prone Himalayan region. Several seismic hazard analyses have been conducted in the Himalayan region, resulting in the development of peak ground acceleration for specific locations. However, challenges exist in using these ground intensities to accurately predict seismic vulnerability. Outdated or macro-level hazard maps, infrequent updates to earthquake databases, lack of expanded prediction equations, generalized GIS databases, data uncertainties, and stochastic-based earthquake catalogs contribute to these challenges. Depth ranges and maximum magnitude evaluations are crucial in seismic hazard assessments. Previous studies used standard or average depth ranges, but this study incorporated appropriate focal depths for point and linear sources. It also utilized a probabilistic approach called Regional Rupture Character (RRC) to estimate maximum magnitudes, setting it apart from studies using different ground motion prediction equations (GMPEs). Furthermore, the study introduced a fully probabilistic technique, called fully probabilistic seismic hazard assessment (FPSHA), to assess ground motion triggering landslides, a novel approach for the region. This study conducted seismic hazard analyses for the Darjeeling Sikkim Himalayan region using three frameworks: deterministic seismic hazard analysis (DSHA), probabilistic seismic hazard analysis (PSHA), and fully probabilistic seismic hazard analysis (FPSHA). DSHA emphasized seismic sources as the primary threat but did not consider uncertainties in the earthquake database and GMPE, potentially impacting results. While deterministic hazard maps evaluate intolerable failure consequences, they lack probabilistic information. PSHA, considering uncertainties in the earthquake database, provides estimates of ground motion exceedance over a specific time period. PSHA ground motions were significantly lower than those from DSHA, possibly due to inclusion of uncertainties. However, PSHA ground motions varied compared to FPSHA, which integrated PSHA with a dynamic slope stability model considering slope properties. FPSHA assessed the most probable ground motion scenarios for landslide triggering over the next 50 years for all slope models. Significant differences in ground motion levels were observed between FPSHA and both DSHA and PSHA, attributed to uncertainties in slope models, GMPE, and seismic source models. It is apparent from this research that seismic landslide hazards can be overestimated or underestimated when relying solely on DSHA and PSHA approaches. While PSHA provides probabilistic ground motions based on historical earthquakes, it is recommended for general seismic infrastructure design. FPSHA, on the other hand, estimates ground motions based on earthquake statistics and soil properties, offering suitable design ground motions for landslide triggering conditions. By considering all possible earthquake scenarios leading to slope instability, FPSHA accounts for specific conditions that can trigger landslides. The updated hazard maps and design charts developed in this study have various applications. They can be utilized in seismic infrastructure design, hazard zoning mapping, landslide monitoring, seismic slope stability analysis, land use planning, code requirements, and implementation of mitigation measures. These outc

Abstract

Autonomous systems perform various tasks across different industries ranging from finance to healthcare to space applications. However, these systems are often deployed in the open world, where it is hard to obtain complete specifications of the objectives and constraints. Operating based on an incomplete model can produce undesired effects, i.e., Negative Side Effects (NSEs). Negative side effects affect the system’s safety and reliability and can be of two types: Markovian and non-Markovian. In this thesis, we try to mitigate negative side effects in environments modeled as Markov decision processes (MDPs). Unlike previous works in this area that associate negative side effects with stateaction pairs, our framework associates them with entire trajectories, which is more general and captures non-Markovian dependence on states and actions. Non-Markovian negative side effects are produced when the agent executes a certain sequence of actions in the deployed environment. Prior works mitigate Markovian negative side effects and can not be easily extended to non-Markovian negative side effects. We build a framework, Controller-Assisted Safe Planning (CASP), for mitigating the non-Markovian negative side effects. Our primary contributions are: 1. We design a model based on Finite State Controllers (FSCs) that can predict the severity of negative side effects for a given trajectory. 2. We learn the model parameters using observed data containing state-action trajectories and the severity of the associated negative side effects. The model is learned such that it generalizes well to unseen data. Information about negative side effects is gathered through Oracle feedback and compactly represented as a finite state controller. 3. We develop a constrained MDP model that uses information from both the underlying MDP and the learned model for planning while avoiding negative side effects. Our empirical evaluation demonstrates the effectiveness of our approach in learning and mitigating Markovian and non-Markovian negative side effects.

Abstract

Document summarization aims to create a precise and coherent summary of a text document. There exist a plethora of deep learning summarization models that are developed mainly for English, which often requires (i) a large training corpus and (ii) efficient pre-trained language models and tools. However, English summarization models for low-resource languages like Indian languages are often limited by rich morphological variation and syntactic and semantic differences. Also, the restricted form of supervision limits the generality and usability of low-resource languages due to the lack of annotated corpora. The Graph Autoencoder (GAE) model has recently shown superior performance on several NLP tasks, even with limited resources. In this work, we propose GAE-ISUMM, an unsupervised Indic Summarization model that extracts summaries. In particular, our proposed model uses GAE and leverages: (i) learning document representations and (ii) jointly learning sentence representations and summary of the document. For evaluation purposes, we introduce TELSUM, a manually annotated summarization dataset comprised of 501 document-summary pairs. Extensive experiments on existing low-resource datasets (XL-Sum) and TELSUM provide the following insights: (i) our proposed model displays state-of-the-art results on XL-Sum and report benchmark results on TELSUM, (ii) Surprisingly, the inclusion of positional and cluster information in the proposed model further improved the performance of summaries. We open-source our dataset and code 1 . On the other hand, with the advancement of various deep-learning methodologies and transformerbased models, summarization has advanced to a new level. However, consistent and standard datasets must be produced to benefit from these deep learning algorithms fully. The creation of dedicated resources is rarely seen for low-resource Indian languages, unlike English which hinders the progress of summarization. To this end, we create summarization resources for Indian languages by introducing ISummCorp (Indic Summarization Corpora) and IndicSumm (Indic Language Summarization Models). IsummCorp is a highly abstractive summarization dataset sourced from the Times Of India (TOI). It is manually annotated by experts across eight Indian languages. Human and intrinsic evaluations demonstrate the high quality, abstraction, and compactness of ISummCorp. IndicSumm is a set of diverse monolingual and multilingual models based on ISummCorp. We refined IndicSumm, by finetuning the sophisticated, multilingual pre-trained mT5 model. With ISummCorp, we show that a model can perform better in a monolingual environment when trained with enough monolingual data than in a multilingual finetuning scenario. To investigate the potential of monolingual models, we finetuned mT5 using ISummCorp in both monolingual and multilingual situations and achieved better performance in a monolingual setting. Furthermore, we compare IndicSumm to other multilingual summarization models (XL-Sum and IndicBART) and achieve the state-of-the-art results.

Abstract

The goal of this thesis is to design a flow synthesis based visual servoing framework enabling longrange obstacle avoidance for Micro Air Vehicles (MAV) flying amongst tall skyscrapers. Recent deep learning based frameworks use optical flow to do high-precision visual servoing. This work explores the question: can a surrogate flow be designed for these high-precision visual-servoing methods which leads to obstacle avoidance? The concept of saliency is explored for identifying high-rise structures in/close to the line of attack amongst other competing skyscrapers and buildings as a collision obstacle. A synthesised flow is used to displace the salient object segmentation mask. This flow is computed in a way that ensures that the visual servoing controller maneuvers the MAV safely around the obstacle. In this approach, a multi-step Cross-Entropy Method (CEM) based servo control is employed to achieve flow convergence, resulting in obstacle avoidance. This novel pipeline is deployed on an MAV to successfully and persistently maneuver amongst high-rises and reach the goal in simulated and photo-realistic realworld scenes. Extensive experimentation is conducted and results are presented to compare the proposed approach with optical flow and short-range depth-based obstacle avoidance methods hence conclusively demonstrating the proposed framework’s merit. Additional visualisations can be found here.

Abstract

This paper proposes a buffer-assisted amplify-and-forward (AF) unmanned aerial vehicle (UAV) relay for communication between two ground nodes without a direct link. These communications are essential in disaster rescue areas where fairness plays a significant role. To achieve fairness, we aim to maximize the minimum information rate. The traditional way of prefixed scheduling the time slots to transfer and receive at the UAV do not guarantee that the communication system uses high signal-to-noise ratio (SNR) communication links instead of low SNR links. Therefore, we choose to employ a buffer at the UAV to store the information and transfer it to the destination nodes in the high SNR links. The pairing of time slots is necessary here since we are using a buffer, and we do not know in which time slot data is transmitted to the destination node after it is received at the UAV. Consequently, we formulate a fairness maximization problem by jointly optimizing the trajectory and power control. Unfortunately, this formulation results in a non-convex problem. We propose a solution based on the principles of the minorize-maximize (MM) algorithm and linear programming relaxation techniques to solve the fairness problem and pairing of time slots. Numerical results demonstrate that the trajectory, power control and paired slots favor the UAV and ground nodes to communicate in the high SNR channel links, thus maintaining fairness.

Abstract

Text-to-speech systems convert any given text to speech. They play a vital role in making Humancomputer interaction (HCI) possible. As humans, we don’t just rely on text (language) to communicate; we use many other mechanisms like voice, gestures, expressions, etc., to communicate efficiently. In natural language processing, vocabulary and grammar tend to take center stage, but those elements of speech only tell half the story. Affective prosody of speech provides larger context and gives meaning to words, and keeps listeners engaged. Current HCI systems largely communicate in text, and they lack a lot of prosodic information, which is crucial in a conversation. To make the HCI systems communicate in speech, text to speech systems should be able to synthesize speech that is expressive and controllable. But the existing text to speech systems learn the average variation in the dataset it’s trained on, which synthesizes samples in a neutral way without many prosodic variations. To this end, we develop a textto-speech system that can synthesize the given emotion where the emotion is represented as a tuple of Arousal, Valance and Dominance (AVD) values. Text to speech systems have a lot of complexities. Training such a system requires the data to be very clear, noiseless, and collecting such data is difficult. If the data is noisy, it will reflect unnecessary artifacts in the synthesized samples. Training emotion based text to speech models is considerably more difficult and not strait forward. The fact that obtaining emotion annotated data for the desired speaker is costly and very subjective makes it a cumbersome task. Current emotion based systems can synthesize emotions with some limitations. (1) Emotion controllability comes at the cost of loss in quality, (2) Have discreet emotions which lack the finer control, and (3) cannot be generalized to new speakers without the annotated emotion data. We propose a system that overcomes the above-mentioned problems by leveraging the largely available corpus of noisy speech annotated with emotions. Even though the data is noisy, our technique trains an emotion based text to speech system that can synthesize desired emotion without any loss of quality in the output. We present a method to control the emotional prosody of Text to Speech (TTS) systems by using phoneme-level intermediate variances/features (pitch, energy, and duration) as levers. We learn how the variances change with respect to emotion. We bring the finer control in the synthesized speech by using AVD values, which can represent emotions in a 3D space. Our proposed method also doesn’t require emotion annotated data for the target speaker. Once trained on the emotion annotated data, it can be applied to any system which has the prediction of the variances as an intermediate step. vi vii With thorough experimental studies, we show that the proposed method improves over the prior art in accurately emulating the desired emotions while retaining the naturalness of speech. We extend the traditional evaluation of using individual sentences for a complete evaluation of HCI systems. We present a novel experimental setup by replacing an actor with a TTS system in offline and live conversations. The emotion to be rendered is either predicted or manually assigned. The results show that the proposed method is strongly preferred over the state-of-the-art TTS system and adds the much-coveted ”human touch” in machine dialogue.