Abstract

Humans can learn multiple languages. If they know a fact in one language, they can answer a question in another language they understand. They can also answer Code-mix (CM) questions: questions which contain both languages. This behavior is attributed to the unique learning ability of humans. Our task aims to study if machines can achieve this. We demonstrate how effectively a machine can answer CM questions. In this work, we adopt a two phase approach: candidate generation and candidate re-ranking to answer questions. We propose a Triplet-Siamese-Hybrid CNN (TSHCNN) to re-rank candidate answers. We show experiments on the SimpleQuestions dataset. Our network is trained only on English questions provided in this dataset and noisy Hindi translations of these questions and can answer English-Hindi CM questions effectively without the need of translation into English. Back-transliterated CM questions outperform their lexical and sentence level translated counterparts by 5% & 35% in accuracy respectively, highlighting the efficacy of our approach in a resource constrained setting.

Abstract

In this paper, we describe Transzaar -an AI powered tool that offers computer aided translation (CAT) functionality: pre-translation analysis,post-editing machine translated content, translation prediction, text aligning, extensive logging and integration with several machine translation (MT) systems. Transzaar aids a human translator to perform various language processing tasks, viz., Translation, Transliteration, Localization, and other kinds of Text Analysis tasks. Using Transzaar, human translators can post-edit the machine translated content, to improve fluency and accuracy of the translated content to match the naturalness of human translation while delivering with better turn-around time.Transzaar aids the process of post-editing, thereby increasing the productivity of human translators by 2-3 folds within couple of months of usage for certain language pairs. It collects feedback continuously, which helps the MT system to further learn and improve periodically, with the additional new generated data-set

Abstract

Code-mixing, use of two or more languages in a single sentence, is ubiquitous; generated by multi-lingual speakers across the world. The phenomenon presents itself prominently in social media discourse. Consequently, there is a growing need for translating code-mixed hybrid language into standard languages. However, due to the lack of gold parallel data, existing machine translation systems fail to properly translate code-mixed text. In an effort to initiate the task of machine translation of code-mixed content, we present a newly created parallel corpus of code-mixed English-Hindi and English. We selected previously available English-Hindi code-mixed data as a starting point for the creation of our parallel corpus. We then chose 4 human translators, fluent in both English and Hindi, for translating the 6088 code-mixed English-Hindi sentences to English. With the help of the created parallel corpus, we analyzed the structure of English-Hindi code-mixed data and present a technique to augment run-of-the-mill machine translation (MT) approaches that can help achieve superior translations without the need for specially designed translation systems. We present an augmentation pipeline for existing MT approaches, like Phrase Based MT (Moses) and Neural MT, to improve the translation of code-mixed text. The augmentation pipeline is presented as a pre-processing step and can be plugged with any existing MT system, which we demonstrate by improving translations done by systems like Moses, Google Neural Machine Translation System (NMTS) and Bing Translator for English-Hindi code-mixed content.

Abstract

Most extractive summarization techniques operate by ranking all the source sentences and then select the top-ranked sentences as the summary. Such methods are known to produce good sum-maries, especially when applied to news articles and scientific texts. However, they do not fareso well when applied to texts such as fictional narratives, which do not have a single central or recurrent theme. This is because usually the information or plot of the story is spread across sev-eral sentences. In this paper, we discuss a different summarization technique called Telegraphic Summarization. Here, we do not select whole sentences, rather pick short segments of text spread across sentences, as the summary. We have tailored a set of guidelines to create such summaries and, using the same, annotate a gold corpus of 200 English short stories.

Abstract

In this paper, we leverage social media platforms such as twitter for developing corpus across multiple languages. The corpus creation methodology is applicable for resource-scarce languages provided the speakers of that particular language are active users onsocial media platforms. We present an approach to extract social media microblogs suchas tweets (Twitter). In this paper, we create corpus for multilingual sentiment analysis and emoji prediction in Hindi, Bengali and Telugu. Further, we perform and analyze multiple NLP tasks utilizing the corpus to get interesting observations

Abstract

A key part of software maintenance is bug reporting and rectifica-tion. Bug reporting is a major issue and due to its asynchronous nature, duplicate bug reporting is common. Detecting duplicate bug reports is an important task in software maintenance in order to avoid the assignment of the same bug to different developers.In this paper, we explore the notion of using word embeddings for retrieving duplicate bug report in large software repositories. We discuss an approach to model each bug report as a dense vector and retrieve its top-k most similar reports for duplicate bug report de-tection. Through experiments on two real world datasets, we show that word embeddings perform better than baselines and related approaches and have the potential to improve duplicate bug report retrieval.

Abstract

Extractive summarization methods operate by ranking and selecting the sentences which best encapsulate the theme of a given document. They do not fare wellin domains like fictional narratives where there is no central theme and core in-formation is not encapsulated by a small set of sentences. For the purpose ofreducing the size of the document while conveying the idea expressed by each sentence, we need more sentence specific methods. Telegraphic summarization,which selects short segments across several sentences, is better suited for such domains. Telegraphic summarization captures the plot better by retaining shorter versions of each sentence while not really concerning itself with grammatically linking these segments. In this paper, we propose an unsupervised deep learning network (NUTS) to generate telegraphic summaries. We use multiple encoder-decoder networks and learn to drop portions of the text that are inferable from thechosen segments. The model is agnostic to both sentence length and style. We demonstrate that the summaries produced by our model show significant quanti-tative and qualitative improvement over those produced by existing methods and baselines.

Abstract

n the past few years, bully and aggressive posts on social media have grown signifi-cantly, causing serious consequences for vic-tims/users of all demographics. Majority of the work in this field has been done for En-glish only. In this paper, we introduce a deep learning based classification system for Face-book posts and comments of Hindi-English Code-Mixed text to detect the aggressive be-haviour of/towards users. Our work focuses ontext from users majorly in the Indian Subcon-tinent. The dataset that we used for our mod-els is provided byTRAC-11in their shared task. Our classification model assigns each Facebook post/comment to one of the threepredefined categories: “Overtly Aggressive”,“Covertly Aggressive” and “Non-Aggressive”.We experimented with 6 classification mod-els and our CNN model on a 10 K-fold cross-validation gave the best result with the predic-tion accuracy of 73.2%

Abstract

In a multilingual or sociolingual configuration Intra-sentential Code Switching (ICS) or Code Mixing (CM) is frequently observed nowadays. In the world most of the people know more than one language. The CM usage is especially apparent in social media platforms. Moreover, ICS is particularly significant in the context of technology, health and law where conveying the upcoming developments are difficult in ones native language. In applications like dialog systems, machine translation, semantic parsing, shallow parsing, etc. CM and Code Switching pose serious challenges. To do any further advancement in code-mixed data, the necessary step is Language Identification. So, in this paper we present a study of various models - Nave Bayes Classifier, Random Forest Classifier, Conditional Random Field (CRF) and Hidden Markov Model (HMM) for Language Identification in English - Telugu Code Mixed Data. Considering the paucity of resources in code mixed languages, we proposed CRF model and HMM model for word level language identification. Our best performing system is CRF-based with an f1-score of 0.91.

Abstract

In this paper, we model the similar question retrieval task as a binary classification problem. We propose a novel approach of “1D-Siamese LSTM for cQA (1D-SLcQA)” to find the semantic similarity between a new question and existing question(s). In 1D-SLcQA, we use a combination of twin LSTM networks and a contrastive loss function to effectively memorize the long term dependencies i.e., capture semantic similarity even when the length of the answers/questions is very large (200 words). The similarity of the questions is modeled using a single network with (1D) (feature) convolution between feature vectors learned from twin LSTM layers. Experiments on large scale real world Yahoo Answers dataset show that 1D-SLcQA outperform the state of the art approach of Siamese cQA approach (SCQA).