Abstract

Data sparsity is a key problem incontemporary neural machine transla-tion (NMT) techniques, especially forresource-scarce language pairs. NMTmodels when coupled with large,high quality parallel corpora providepromising results and are an emergingalternative to phrase-based StatisticalMachine Translation (SMT) systems.A solution to overcome data sparsitycan facilitate leveraging of NMT mod-els across language pairs, thereby pro-viding high quality translations de-spite the lack of large parallel cor-pora. In this paper, we demonstrate athree-phase integrated approach whichcombines weakly supervised and semi-supervised learning with NMT tech-niques to build a robust model usinga limited amount of parallel data. Weconduct experiments for five languagepairs (thereby generating ten systems)and our results show a substantial in-crease in translation quality over abaseline NMT model trained only

Abstract

Neural Machine Translation is emerg-ing as the de facto standard for Ma-chine Translation across the globe.Statistical Machine Translation hasbeen the state-of-the-art for translationamong Indian languages. This paperprobes into the effectiveness of NMTfor Indian languages and compares thestrengths and weaknesses of NMT withSMT through a vis-a-vis qualitative es-timation on different linguistic param-eters. We compare the outputs ofboth models for the languages English,Malayalam and Hindi; and test themon various linguistic parameters. Weconclude that NMT works better inmost of the settings, however there isstill immense scope for the bettermentof accuracy for translation of IndianLanguages. We describe the challengesfaces especially when dealing with lan-guages from different language families

Abstract

We present an approach for POS tag-ging with out any labeled data. Ourmethod requires translated sentencesfrom a pair of languages. We usedfeature transfer from a resource richlanguage to resource poor languages.Across 8 different Indian Languages,we achieved encouraging accuracieswithout any knowledge of the targetlanguage and any human annotation.This will help us in creating annotatedcorpora for

Abstract

Sentiment analysis in a resource scarcelanguage is a tedious task. We pro-pose a novel method for transfer learn-ing from a target language to English.Our system doesn’t rely on labeleddata for the target language but in-stead links itself onto already existingand extensively labeled word-level lex-ical resource in English (ESWN1) anda semantic parser. Our proposed sys-tem transparently needs no target lan-guage sentiment corpus, and exploitscomplex linguistic structure of the tar-get language for sentiment prediction.This cross lingual approach gives netaccuracy as 83.6%, an improvement of5.4% over the baseline system.

Abstract

n this paper, we present a generic semi-supervised Word Sense Disambiguation(WSD) method. Currently, the existingWSD methods extensively use domain re-sources and linguistic knowledge.Ourproposed method extractscontext basedlistsfrom a small sense-tagged and un-tagged training data without using do-main knowledge. Experiments on Hindiand Marathi Tourism and Health domainsshow that it gives good performance with-out using any language specific linguisticinformation except the sense IDs presentin the sense-tagged training set and workswell even with small training data by han-dling the data sparsity issue. Other ad-vantages are that domain expertise is notneeded for crafting and selecting featuresto build the WSD model and it can handlethe problem of non availability of match-ing contexts in sense-tagged training set.It also finds sense IDs of those test wordswhich are not present in sense-tag

Abstract

We investigate the problem of parsing conversational data of morphologically-rich languages such as Hindi where argument scrambling occurs frequently. We evaluate a state-of-the-art non-linear transition-based parsing system on a new dataset containing 506 dependency trees for sentences from Bollywood (Hindi) movie scripts and Twitter posts of Hindi monolingual speakers. We show that a dependency parser trained on a newswire treebank is strongly biased towards the canonical structures and degrades when applied to conversational data. Inspired by Transformational Generative Grammar (Chomsky, 1965), we mitigate the sampling bias by generating all theoretically possible alternative word orders of a clause from the existing (kernel) structures in the treebank. Training our parser on canonical and transformed structures improves performance on conversational data by around 9% LAS over the baseline newswire parser.

Abstract

With immense amount of data growing on web in Hindi, a text summariser would be helpful in summarising Government data, medical reports, news, and research articles. Hindi is the fourth most-spoken first language in the world. Hindi written in the Devanagari script is the official language of the Government of India. There is no public dataset for extractive summarisation available in Hindi and thus a dataset of 24253 News articles was extracted and the extractive summaries results were evaluated on various parameters with manual gold summaries of exactly 60 words each.

Abstract

Extractive summarization techniques typically aim to maximize the information coverage of the summary with respect to the original corpus and report accuracies in ROUGE scores. Automated text summarization techniques should consider the dimensions of comprehensibility, coherence and readability. In the current work, we identify the discourse structure which provides the context for the creation of a sentence. We leverage the information from the structure to frame a monotone (non-decreasing) sub-modular scoring function for generating comprehensible summaries. Our approach improves the overall quality of comprehensibility of the summary in terms of human evaluation and gives sufficient content coverage with comparable ROUGE score. We also formulate a metric to measure summary comprehensibility in terms of Contextual Independence of a sentence. The metric is shown to be representative of human judgement of text comprehensibility.

Abstract

n this study, the problem of shallow parsing of Hindi-English code-mixed social media text (CSMT) has been addressed. We have annotated the data, developed a language identifier, a normalizer, a part-of-speech tagger and a shallow parser. To the best of our knowledge, we are the first to attempt shallow parsing on CSMT. The pipeline developed has been made available to the research community with the goal of enabling better text analysis of Hindi English CSMT. The pipeline is accessible at this http URL.

Abstract

We present Kathaa, an Open Source web-based Visual Programming Framework for Natural Language Processing (NLP) Systems. Kathaa supports the design, execution and analysis of complex NLP systems by visually connecting NLP components from an easily extensible Module Library. It models NLP systems an edge-labeled Directed Acyclic MultiGraph, and lets the user use publicly co-created modules in their own NLP applications irrespective of their technical proficiency in Natural Language Processing. Kathaa exposes an intuitive web based Interface forthe users to interact with and modify complex NLP Systems; and a precise Module definition API to allow easy integration of new state of the art NLP components. Kathaa enables researchers to publish their services in a standardized format to enable the masses to use their services out ofthe box. The vision of this work is to pave the way for a system like Kathaa, to be the Lego blocks of NLP Research and Applications. As a practical use case we use Kathaa to visually implement the Sampark Hindi-Panjabi Machine Translation Pipeline and the Sampark Hindi-Urdu Machine Translation Pipeline, to demonstrate the fact that Kathaa can handle really complex NLP systems while still being intuitive for the end user.