Abstract

In this paper, we describe Transzaar -an AI powered tool that offers computer aided translation (CAT) functionality: pre-translation analysis,post-editing machine translated content, translation prediction, text aligning, extensive logging and integration with several machine translation (MT) systems. Transzaar aids a human translator to perform various language processing tasks, viz., Translation, Transliteration, Localization, and other kinds of Text Analysis tasks. Using Transzaar, human translators can post-edit the machine translated content, to improve fluency and accuracy of the translated content to match the naturalness of human translation while delivering with better turn-around time.Transzaar aids the process of post-editing, thereby increasing the productivity of human translators by 2-3 folds within couple of months of usage for certain language pairs. It collects feedback continuously, which helps the MT system to further learn and improve periodically, with the additional new generated data-set

Abstract

Code-mixing, use of two or more languages in a single sentence, is ubiquitous; generated by multi-lingual speakers across the world. The phenomenon presents itself prominently in social media discourse. Consequently, there is a growing need for translating code-mixed hybrid language into standard languages. However, due to the lack of gold parallel data, existing machine translation systems fail to properly translate code-mixed text. In an effort to initiate the task of machine translation of code-mixed content, we present a newly created parallel corpus of code-mixed English-Hindi and English. We selected previously available English-Hindi code-mixed data as a starting point for the creation of our parallel corpus. We then chose 4 human translators, fluent in both English and Hindi, for translating the 6088 code-mixed English-Hindi sentences to English. With the help of the created parallel corpus, we analyzed the structure of English-Hindi code-mixed data and present a technique to augment run-of-the-mill machine translation (MT) approaches that can help achieve superior translations without the need for specially designed translation systems. We present an augmentation pipeline for existing MT approaches, like Phrase Based MT (Moses) and Neural MT, to improve the translation of code-mixed text. The augmentation pipeline is presented as a pre-processing step and can be plugged with any existing MT system, which we demonstrate by improving translations done by systems like Moses, Google Neural Machine Translation System (NMTS) and Bing Translator for English-Hindi code-mixed content.

Abstract

Most extractive summarization techniques operate by ranking all the source sentences and then select the top-ranked sentences as the summary. Such methods are known to produce good sum-maries, especially when applied to news articles and scientific texts. However, they do not fareso well when applied to texts such as fictional narratives, which do not have a single central or recurrent theme. This is because usually the information or plot of the story is spread across sev-eral sentences. In this paper, we discuss a different summarization technique called Telegraphic Summarization. Here, we do not select whole sentences, rather pick short segments of text spread across sentences, as the summary. We have tailored a set of guidelines to create such summaries and, using the same, annotate a gold corpus of 200 English short stories.

Abstract

In this paper, we leverage social media platforms such as twitter for developing corpus across multiple languages. The corpus creation methodology is applicable for resource-scarce languages provided the speakers of that particular language are active users onsocial media platforms. We present an approach to extract social media microblogs suchas tweets (Twitter). In this paper, we create corpus for multilingual sentiment analysis and emoji prediction in Hindi, Bengali and Telugu. Further, we perform and analyze multiple NLP tasks utilizing the corpus to get interesting observations

Abstract

A key part of software maintenance is bug reporting and rectifica-tion. Bug reporting is a major issue and due to its asynchronous nature, duplicate bug reporting is common. Detecting duplicate bug reports is an important task in software maintenance in order to avoid the assignment of the same bug to different developers.In this paper, we explore the notion of using word embeddings for retrieving duplicate bug report in large software repositories. We discuss an approach to model each bug report as a dense vector and retrieve its top-k most similar reports for duplicate bug report de-tection. Through experiments on two real world datasets, we show that word embeddings perform better than baselines and related approaches and have the potential to improve duplicate bug report retrieval.

Abstract

Extractive summarization methods operate by ranking and selecting the sentences which best encapsulate the theme of a given document. They do not fare wellin domains like fictional narratives where there is no central theme and core in-formation is not encapsulated by a small set of sentences. For the purpose ofreducing the size of the document while conveying the idea expressed by each sentence, we need more sentence specific methods. Telegraphic summarization,which selects short segments across several sentences, is better suited for such domains. Telegraphic summarization captures the plot better by retaining shorter versions of each sentence while not really concerning itself with grammatically linking these segments. In this paper, we propose an unsupervised deep learning network (NUTS) to generate telegraphic summaries. We use multiple encoder-decoder networks and learn to drop portions of the text that are inferable from thechosen segments. The model is agnostic to both sentence length and style. We demonstrate that the summaries produced by our model show significant quanti-tative and qualitative improvement over those produced by existing methods and baselines.

Abstract

n the past few years, bully and aggressive posts on social media have grown signifi-cantly, causing serious consequences for vic-tims/users of all demographics. Majority of the work in this field has been done for En-glish only. In this paper, we introduce a deep learning based classification system for Face-book posts and comments of Hindi-English Code-Mixed text to detect the aggressive be-haviour of/towards users. Our work focuses ontext from users majorly in the Indian Subcon-tinent. The dataset that we used for our mod-els is provided byTRAC-11in their shared task. Our classification model assigns each Facebook post/comment to one of the threepredefined categories: “Overtly Aggressive”,“Covertly Aggressive” and “Non-Aggressive”.We experimented with 6 classification mod-els and our CNN model on a 10 K-fold cross-validation gave the best result with the predic-tion accuracy of 73.2%

Abstract

In a multilingual or sociolingual configuration Intra-sentential Code Switching (ICS) or Code Mixing (CM) is frequently observed nowadays. In the world most of the people know more than one language. The CM usage is especially apparent in social media platforms. Moreover, ICS is particularly significant in the context of technology, health and law where conveying the upcoming developments are difficult in ones native language. In applications like dialog systems, machine translation, semantic parsing, shallow parsing, etc. CM and Code Switching pose serious challenges. To do any further advancement in code-mixed data, the necessary step is Language Identification. So, in this paper we present a study of various models - Nave Bayes Classifier, Random Forest Classifier, Conditional Random Field (CRF) and Hidden Markov Model (HMM) for Language Identification in English - Telugu Code Mixed Data. Considering the paucity of resources in code mixed languages, we proposed CRF model and HMM model for word level language identification. Our best performing system is CRF-based with an f1-score of 0.91.

Abstract

In this paper, we model the similar question retrieval task as a binary classification problem. We propose a novel approach of “1D-Siamese LSTM for cQA (1D-SLcQA)” to find the semantic similarity between a new question and existing question(s). In 1D-SLcQA, we use a combination of twin LSTM networks and a contrastive loss function to effectively memorize the long term dependencies i.e., capture semantic similarity even when the length of the answers/questions is very large (200 words). The similarity of the questions is modeled using a single network with (1D) (feature) convolution between feature vectors learned from twin LSTM layers. Experiments on large scale real world Yahoo Answers dataset show that 1D-SLcQA outperform the state of the art approach of Siamese cQA approach (SCQA).

Abstract

Movies are one of the most prominent means of entertainment. The widespread use of the Internet in recent times has led to large volumes of data related to movies being generated and shared online. People often prefer to express their views online in English as compared to other local languages. This leaves us with a very little amount of data in languages apart from English to work on. To overcome this, we created the Multi-Language Movie Review Dataset (MLMRD). The dataset consists of genre, rating, and synopsis of a movie across multiple languages, namely Hindi, Telugu, Tamil, Malayalam, Korean, French, and Japanese. The genre of a movie can be identified by its synopsis. Though the rating of a movie may depend on multiple factors like the performance of actors, screenplay, direction etc but in most of the cases, synopsis plays a crucial role in the movie rating. In this work, we provide various model architectures that can be used to predict the genre and the rating of a movie across various languages present in our dataset based on the synopsis