Abstract

Foundational Language Models perform significantly better on downstream tasks in specialised domains (such as law, computer science, and medical science) upon being further pre-trained on extensive domain-specific corpora, but this continual pre-training incurs heavy computational costs. Indeed, some of the most performant specialised language models such as BioBERT incur even higher computing costs during domain-specific training than the pre-training cost of the foundational models they are initialised from. In this paper, we argue that much of the extended pre-training is redundant, with models seemingly wasting valuable resources re-learning lexical and semantic patterns already well-represented in their foundational models such as BERT, T5 and GPT. Focusing on Masked Language Models, we introduce a novel domain-specific masking strategy that is designed to facilitate continual learning while minimizing the training cost. Using this approach, we train and present a BERT-based model trained on a biomedical corpus that matches or surpasses traditionally trained biomedical language models in performance across several downstream classification tasks while incurring up to 11 times lower training costs.

Abstract

Foundational Language Models perform significantly better on downstream tasks in the biomedical domain upon being further pre-trained on extensive biomedical corpora, but this continual pre-training incurs heavy computational costs. Indeed, some of the most performant biomedical language models incur even more computing costs during domain-specific training than the entire training cost of the foundational models they are warm-started from. In this paper, we argue that much of the extended pre-training is redundant, with models seemingly wasting valuable resources re-learning lexical and semantic patterns already well-represented in their foundational models such as BERT, T5 and GPT. Focusing on Masked Language Models, we introduce a novel domain-specific masking strategy that is designed to facilitate continual learning while minimizing the training cost. Using this approach, we train and present a BERT-based model that matches or surpasses traditionally trained biomedical language models in performance across several downstream classification tasks while incurring up to 11 times lower training costs.

Abstract

According to the cooperative principle (Grice 1975), utterances in a conversation adhere to the four maxims of conversational implicature – quantity, relation, quality and manner. An implicature arises when a maxim is flouted or contradicted, and the speaker is forced to infer a meaning. Consider the example: Speaker A: का िदलीप को परीकाएं आसान लग रही है? Kya Dilip ko parikshayein aasan lag rahi hain? (Does Dilip find the exam easy?) Speaker B: आज-कल वो सो नही पा रहा | [Aaj-kal wo so nahi pa raha.] (He hasn’t been able to sleep lately.) Speaker B flouts the maxim of relevance by not answering the question with a yes or no. Instead, his reply that Dilip has trouble sleeping implies that he doesn’t find the exams to be easy. While computational approaches have made some progress in English implicature benchmarks - GRICE (Zheng, 2021), IMPRESS (Jeretic 2020) and Conversational Implicature benchmark (CIB) (George and Mamidi, 2020), there are no benchmarks for Hindi. Hence, we propose a Hindi implicature benchmark- Indirect Questions Hi Implicature Benchmark - Toledo-Ronen (2020) found that translation- based methods lose nuances in argument evaluation. XCOPA benchmark (Ponti 2020) & Huang (2024) demonstrated that translated data hurts model performance for structurally divergent non- English languages. Hindi is a free word order, morphologically rich language. It is structurally different from English hence, translating English implicature benchmarks may not be unsuitable. Wang (2024) present indirect answers to yes-no questions in dialogues benchmark. Similar to their approach, we plan to collect Hindi interviews and extract indirect questions along with their answers involving implicature. We will annotate 5000 such questions. We will maintain a constraint that previous dialogues will be added to the question as a background to simplify the inference over a single dialogue turn. Evaluation will rely on ROUGE scores to measure correctness, as we leave exploration of suitable metrics for future work. For model evaluation, we plan to leverage a) English LLMs (meta-llama/Llama-3.3-70B (Grattafiori 2024) and the current state-of-the-art reported by Sravanthi (2024), FlanT5-XXL (Chung 2022) finetuned on Hindi translated GRICE (Zheng 2021), IMPRESS (Jeretic 2020) and CIB (George and Mamidi 2020) and b) Hindi LLMs (Llama-3-Nanda-10B-Chat (Choudhury 2024) and Airavata (Gala 2024)).

Abstract

Computational Linguistics is an interdisciplinary field of computer science and linguistics that focuses on designing computational models and algorithms for processing, analyzing, and generating human language. Over recent years, this field has made substantial progress. While its primary emphasis tends to center around widely spoken languages, there is equal importance in investigating languages that are not commonly spoken but have contributed immensely to the literature, culture, and philosophy of the society. Thus, this survey article comprehensively delves into the exploration of computational tasks undertaken for Sanskrit, an ancient language of the Indian sub-continent steeped in a wealth of literary heritage. The purpose of this study is to provide an overview of the progress made thus far in the computational analysis of Sanskrit, while also reviewing the current digital infrastructure that supports these efforts. Additionally, our study also identifies potential avenues for future research, serving as a reference for anyone interested in advancing their exploration in this field.

Abstract

There is a high prevalence of disinformation and manipulative narratives in online new sources today, and verification of its informative integrity is a vital need as online audience is highly susceptible to being affected by such propaganda or disinformation. The task of verifying any online information is, however, a significant challenge. The task Multilingual Characterization and Extraction of Narratives from Online News, therefore focuses on developing novel methods of analyzing news ecosystems and detecting manipulation attempts to address this challenge. As a part of this effort, we focus on the subtask of Entity Framing, which involves assigning named entities in news articles one of three main roles ( Protagonist, Antagonist, and Innocent) with a further fine-grained role distinction. We propose a pipeline that involves summarizing the article with the summary being centered around the entity. The entity and its entity-centric summary is then used as input for a BERT-based classifier to carry out the final role classification. Finally, we experiment with different approaches in the steps of the pipeline and compare the results obtained by them.

Abstract

This paper explores the concept of solving implicature in Natural Language Processing (NLP), highlighting its significance in understanding indirect communication. Drawing on foundational theories by Austin, Searle, and Grice, we discuss how implicature extends beyond literal language to convey nuanced meanings. We review existing datasets, including the Pragmatic Understanding Benchmark (PUB), that assess models’ capabilities in recognizing and interpreting implicatures. Despite recent advances in large language models (LLMs), challenges remain in effectively processing implicature due to limitations in training data and the complexities of contextual interpretation. We propose future directions for research, including the enhancement of datasets and the integration of pragmatic reasoning tasks, to improve LLMs’ understanding of implicature and facilitate better human-computer interaction.

Abstract

In today’s digital age, access to knowledge and information is crucial for societal growth. Although widespread resources like Wikipedia exist, there is still a linguistic barrier to breakdown for low-resource languages. In India, millions of individuals still lack access to reliable information from Wikipedia because they are only proficient in their regional language. To address this gap, our work focuses on enhancing the content and digital footprint of multiple Indian languages. The primary objective of our work is to improve knowledge accessibility by generating a substantial volume of high-quality Wikipedia articles in Telugu, a widely spoken language in India with around 95.7 million native speakers. Our work aims to create Wikipedia articles and also ensures that each article meets necessary quality standards such as a minimum word count, inclusion of images for reference, and an infobox. Our work also adheres to the five core principles of Wikipedia. We streamline our article generation process, leveraging NLP techniques such as translation, transliteration, and template generation and incorporating human intervention when necessary. Our contribution is a collection of 8,929 articles in the movie domain, now ready to be published on Telugu Wikipedia.

Abstract

Open Vocabulary Keyword Spotting is essential in numerous applications, from virtual assistants to security systems, as it allows systems to identify specific words or phrases in continuous speech. In this paper, we propose a novel end-to-end method for detecting user-defined open vocabulary keywords by leveraging linguistic patterns for the correlation between audio and text modalities. Our approach utilizes quantized pre-trained foundation models for robust audio embeddings and a unique lightweight Multi-Scale Linear Attention (MSLA) network that aligns speech and text representations for effective cross-modal agreement. We evaluate our method on two distinct datasets, comparing its performance against other baselines. The results highlight the effectiveness of our approach, achieving significant improvements over the Cross-Modality Correspondence Detector (CMCD) method, with a 16.08% increase in AUC and a 17.2% reduction in EER metrics on the Google Speech Commands dataset. These findings demonstrate the potential of our method to advance keyword spotting across various real-world applications.

Abstract

Coreference resolution, the process of determining what a referring expression (a pronoun or a noun phrase) refers to in discourse, is a critical aspect of natural language understanding. However, the development of computational models for coreference resolution in low-resource languages, such as the Dravidian (and more broadly all South Asian) languages, still remains a significant challenge due to the scarcity of annotated corpora in these languages. To address this data scarcity, we adopt a pipeline that translates the English GAP dataset into various South Asian languages, creating a multi-lingual coreference dataset mGAP. Our research aims to leverage this dataset and develop two novel models, namely the joint embedding model and the cross attention model for coreference resolution with Dravidian languages in mind. We also demonstrate that cross-attention captures pronoun-candidate relations better leading to improved coreference resolution. We also harness the similarity across South Asian languages via transfer learning in order to use high resource languages to learn coreference for low resource languages.

Abstract

In recent studies, the extensive utilization of large language models has underscored the im- portance of robust evaluation methodologies for assessing text generation quality and rel- evance to specific tasks. This has revealed a prevalent issue known as hallucination, an emergent condition in the model where gener- ated text lacks faithfulness to the source and deviates from the evaluation criteria. In this study, we formally define hallucination and pro- pose a framework for its quantitative detection in a zero-shot setting, leveraging our definition and the assumption that model outputs entail task and sample specific inputs. In detecting hallucinations, our solution achieves an accu- racy of 0.78 in a model-aware setting and 0.61 in a model-agnostic setting. Notably, our so- lution maintains computational efficiency, re- quiring far less computational resources than other SOTA approaches, aligning with the trend towards lightweight and compressed models.