Abstract

Code-switching is a phenomenon of mixing grammatical structures of two or more languages under varied social constraints. The code-switching data differ so radically from the benchmark corpora used in NLP community that the application of standard technologies to these data degrades their performance sharply. Unlike standard corpora, these data often need to go through additional processes such as language identification, normalization and/or back-transliteration for their efficient processing. In this paper, we investigate these indispensable processes and other problems associated with syntactic parsing of code-switching data and propose methods to mitigate their effects. In particular, we study dependency parsing of code-switching data of Hindi and English multilingual speakers from Twitter. We present a treebank of Hindi-English code-switching tweets under Universal Dependencies scheme and propose a neural stacking model for parsing that efficiently leverages part-of-speech tag and syntactic tree annotations in the code-switching treebank and the preexisting Hindi and English treebanks. We also present normalization and back-transliteration models with a decoding process tailored for code-switching data. Results show that our neural stacking parser is 1.5% LAS points better than the augmented parsing model and our decoding process improves results by 3.8% LAS points over the first-best normalization and/or backtransliteration.

Abstract

The Part-Of-Speech tag of a word can provide crucial information for a large number of tasks, and so, it is of utmost importance that the POS tagged data is accurate. However, manually checking the data is a tedious and time consuming task. Thus, there is a need for an Automatic Error Correction and Validation model for any POS Tagged Data. In this paper,we work towards achieving the aforementioned goal for Hindi POS Tagging. This is achieved by using an ensemble model consisting of three POS Tagging Models. Based on the predictions made by the three models, and the POS tag present in the dataset, the ensemble model predicts the presence of an error. The POS tagging models explored were the Hidden Markov Model, Support Vector Machine, Conditional Random Fields, Long Short Term Memory (LSTM) Networks, Bidirectional LSTM Networks, and Logistic Regression. A Fully Connected Neural Network was used to build the ensemble model, and it achieved an accuracy of 94.02%.

Abstract

Automatic Word problem solving has always posed a great challenge for the NLP community. Usually a word problem is a narrative comprising of a few sentences and a question is asked about a quantity referred in the sentences. Solving word problem involves reasoning across sentences, identification of operations, their order, relevant quantities and discarding irrelevant quantities. In this paper, we present a novel approach for automatic arithmetic word problem solving. Our approach starts with frame identification. Each frame can either be classified as a state or an action frame. The frame identification is dependent on the verb in a sentence. Every frame is unique and is identified by its slots. The slots are filled using dependency parsed output of a sentence. The slots are entity holder, entity, quantity of the entity, recipient, additional information like place, time. The slots and frames helps to identify the type of question asked and the entity referred. Action frames act on state frame(s) which causes a change in quantities of the state frames. The frames are then used to build a graph where any change in quantities can be propagated to the neighboring nodes. Most of the current solvers can only answer questions related to the quantity, while our system can answer different kinds of questions like ‘who’, ‘what’ other than the quantity related questions ‘how many’. There are three major contributions of this paper. 1. Frame Annotated Corpus (with a frame annotation tool) 2. Frame Identification Module 3. A new easily understandable Framework for word problem solving

Abstract

POS Tagging serves as a preliminary task for many NLP applications. Kannada is a relatively poor Indian language with very limited number of quality NLP tools available for use. An accurate and reliable POS Tagger is essential for many NLP tasks like shallow parsing, dependency parsing, sentiment analysis, named entity recognition. We present a statistical POS tagger for Kannada using different machine learning and neural network models. Our Kannada POS tagger outperforms the state-of-the-art Kannada POS tagger by 6%. Our contribution in this paper is three folds - building a generic POS Tagger, comparing the performances of different modeling techniques, exploring the use of character and word embeddings together for Kannada POS Tagging.

Abstract

Ensemble POS taggers are a good choice to integrate and leverage benefits of various types of POS taggers. This can help the large number (6500+) of resource-poor languages which do not have much annotated training data by providing ways to integrate semi-supervised/unsupervised taggers with supervised taggers. In this paper we present our experiments of developing ensemble POS taggers using a decision tree. We integrate a semi-supervised data mining approach that uses context based lists (CBLs) for POS tagging with supervised (1) Support Vector Machine based POS tagger, called SVMTool and (2) Conditional Random Field based POS tagger. The results are enhanced semi-supervised ensemble POS taggers which outperform the base methods. In these POS taggers, we use a decision tree to decide when to rely on the output of supervised tagger, and when to rely on the semi-supervised CBL method. The CBL based tagger uses rich contextual information which helps in tagging both existing and unseen words and uses no domain knowledge while supervised taggers give good performance for words present in the training model and can include domain based features. Hence, these algorithms have complementary strengths and in our ensemble we are able to combine these strengths. Enhanced performance of our new POS taggers over the base methods suggests that integrating these methods combines the qualities of these in the new tagger which enhances the performance. Therefore, these new semi-supervised ensemble taggers are more suitable for resource-poor languages.

Abstract

This paper describes the systems submitted by IIT (BHU), Varanasi/IIIT Hyderabad (IITBHU–IIITH) for Task 1 of CoNLL– SIGMORPHON 2018 Shared Task on Universal Morphological Reinflection (Cotterell et al., 2018). The task is to generate the inflected form given a lemma and set of morphological features. The systems are evaluated on over 100 distinct languages and three different resource settings (low, medium and high). We formulate the task as a sequence to sequence learning problem. As most of the characters in inflected form are copied from the lemma, we use Pointer-Generator Network (See et al., 2017) which makes it easier for the system to copy characters from the lemma. PointerGenerator Network also helps in dealing with out-of-vocabulary characters during inference. Our best performing system stood 4th among 28 systems, 3rd among 23 systems and 4th among 23 systems for the low, medium and high resource setting respectively.

Abstract

This paper presents DILTON, a system which solves simple arithmetic word problems. DILTON first predicts the operation that is to be performed (’-’,’+’,’*’,’/’) through a deep neural network based model and then uses it to generate the answer. DILTON divides the question into two parts - world state and query as shown in Figure 1. The world state and the query are processed separately in two different networks and finally the networks are merged to predict the final operation. DILTON learns to predict operations with 8.81 % in a corpus of primary school questions. With simple similarity between the contexts of quantities appearing in the problem and the question text, we are able to identify 92.25 % of relevant quantities and solve 81% of the questions. Our code and data is publicly available

Abstract

Hindi and Urdu are spoken primarily in northern India and Pakistan; together, they constitute the third largest language spoken in the world. 1 They are two standard- ized registers of what has been called the Hindustani language, which belongs to the Indo-Aryan language family. Masica [1993] states that, while they are different languages officially, they are not even different dialects or subdialects in a linguistic sense. Rather, they are different literary styles based on the same linguistically defined

Abstract

Thispaperpresentsourworktoapplynonlinearneuralnetworkforparsingfiver esourcep oorI ndianL anguagesbe-longingtotwomajorlanguagefamilies-Indo-AryanandDravidian.BengaliandMarathiareIndo-AryanlanguageswhereasKannada,TeluguandMalayalambelongtotheDravidianfamily.WhilelittleworkhasbeendonepreviouslyonBengaliandTelugulineartransition-basedparsing,wepresentoneofthefirstparsersforMarathi,KannadaandMalayalam.AlltheIndianlanguagesarefreewordorderandrangefrombeingmoderatetoveryrichinmorphology.Thereforeinthisworkweproposetheusageoflinguisticallymo-tivatedmorphologicalfeatures(suffixandpostposition)inthenonlinearframework,tocapturetheintricaciesofboththelan-guagefamilies.Wealsocapturechunkandgender,number,personinformationelegantlyinthismodel.Weputforwardwaystorepresentthesefeaturescosteffe

Abstract

In this paper, we propose efficient and less resource-intensive strategies for parsing of code-mixed data. These strategies are not constrained by in-domain annotations, rather they leverage pre-existing monolingual annotated resources for training. We show that these methods can produce significantly better results as compared to an informed baseline. Due to lack of an evaluation set for code-mixed structures, we also present a data set of 450 Hindi and English code-mixed tweets of Hindi multilingual speakers for evaluation.