Abstract

Product reviews on e-commerce platforms are a critical form of user-generated content that influence consumer decisions. However, these reviews are predominantly in English, creating a significant accessibility barrier for users who are not fluent in English. When translating into major Indian languages using the current models, the outputs often fail to capture domain-specific features and colloquial style, resulting in stylistically unnatural texts. To address this gap, we introduce textbf{STAR-IL}, a human-annotated, multilingual, parallel corpus for style-aware translation of product reviews. We evaluate the performance of several state-of-the-art models on our dataset for the task of product review translation. Our experiments show that models fine-tuned on STAR-IL achieve significant average performance gain of textbf{5.77} points in BLEU and textbf{3.78} points in COMET, when compared to their baselines, across all languages. Our dataset provides a valuable benchmark for future research in style-aware product review translation. The STAR-IL dataset is publicly available at https://github.com/ltrc/STAR-IL-Corpus

Abstract

The extent to which large language models (LLMs) can perform culturally grounded reasoning across non-English languages remains underexplored. This paper examines the reasoning and self-assessment abilities of LLMs across seven major Indian languages- Bengali, Gujarati, Hindi, Kannada, Malayalam, Tamil, and Telugu. We introduce a multilingual riddle dataset combining traditional riddles with context-reconstructed variants and evaluate five LLMs- Gemini 2.5 Pro, Gemini 2.5 Flash, Mistral-Saba, LLaMA-4-Scout, and LLaMA-4-Maverick under seven prompting strategies. In the first stage, we assess riddle-solving performance and find that while Gemini 2.5 Pro performs best overall, few-shot methods yield only marginal gains, and accuracy varies notably across languages. In the second stage, we conduct a self-evaluation experiment to measure reasoning consistency. The results reveal a key finding: a model's initial accuracy is inversely correlated with its ability to identify its own mistakes. Top-performing models such as Gemini 2.5 Pro are overconfident (4.34% True Negative Rate), whereas lower-performing models like LLaMA-4-Scout are substantially more self-aware (42.09% True Negative Rate). These results point to clear gaps in multilingual reasoning and highlight the need for models that not only reason effectively but also recognize their own limitations.

Abstract

This paper details the system submitted to the
RANLP-MCR-IL 2025 Shared Task on coref-
erence resolution across five Indian languages:
Hindi, Kannada, Malayalam, Tamil, and Tel-
ugu. Coreference resolution involves identify-
ing expressions in a text that refer to the same
entity. Our submission is based on the appli-
cation of a state-of-the-art end-to-end neural
coreference resolution model, which uses the
bert-base-multilingual-cased pre-trained model
for its powerful cross-lingual representations.
The system is trained from scratch using the
task-provided data and evaluated using the stan-
dard CoNLL coreference metrics: B-CUBED
(BCUB), CEAFM, and CEAFE on the official
test sets. We followed the span-based architec-
ture with higher-order refinement and trained
all models under the same setup across lan-
guages. Preprocessing involved converting
token-level annotations into CoNLL 2012 for-
mat. To maintain fairness across settings, we
applied the same hyperparameters and prepro-
cessing steps to all language datasets. Our
system achieves strong performance, including
an average F1 score of 49.45% for Kannada,
47.11% for Telugu, 37.77% for Hindi, 37.42%
for Malayalam, and 37.06% for Tamil.

Abstract

We study Gemma-2-2B on a controlled role-gated retrieval task where a prepositional gate (to or from) selects which of two entities is correct. On 60 single-token name pairs the model attains 100% accuracy with a mean flip magnitude (~3.5) (sum of per-condition correctness margins). Using causal tracing, we identify a Query-Gated Courier circuit with three stages: (1) a gate token (from/to) writes a role feature at the answer; (2) this feature perturbs late-layer courier queries, shifting their (q · k) preference; (3) couriers attend to the correct name and inject it via OV, raising its logit. Gate-residual swaps flip predictions, and a compact nine-head keep set reproduces the behavior with high fidelity. The circuit gives a potential algorithm for role tracking and aligns with the Paninian Karaka analysis, mapping to to sampradana and from to apadana.

Abstract

Developing robust Natural Language Understanding (NLU) for morphologically rich Dravidian languages like Kannada, Malayalam, Tamil, and Telugu presents significant challenges due to their agglutinative nature and syntactic complexity. In this work, we present the Dravidian NLP Suite tackling five core tasks: Morphological Analysis (MA), POS Tagging (POS), Named Entity Recognition (NER), Dependency Parsing (DEP), and Coreference Resolution (CR), trained for monolingual models and multilingual models. To facilitate this, we present the Dravida dataset, meticulously annotated multilingual corpus for these tasks across all four languages. Our experiments demonstrate that a multilingual model, which utilizes shared linguistic features and cross-lingual patterns inherent to the Dravidian family, consistently outperforms its monolingual counterparts across all tasks. These findings suggest that multilingual learning is an effective approach for enhancing Natural Language Understanding (NLU) capabilities, particularly for languages belonging to the same family. To the best of our knowledge, this is the first work to jointly address all these core tasks on the Dravidian languages.

Abstract

Large multilingual transformers such as XLM-RoBERTa achieve impressive performance on diverse NLP benchmarks, but understanding how they internally encode grammatical information remains challenging. This study investigates the encoding of syntactic and morphological information derived from the Paninian grammatical framework--specifically designed for morphologically rich Indian languages--across model layers. Using diagnostic probing, we analyze the hidden representations of frozen XLM-RoBERTa-base, mBERT, and IndicBERT models across seven Indian languages (Hindi, Kannada, Malayalam, Marathi, Telugu, Urdu, Bengali). Probes are trained to predict Paninian dependency relations (by edge probing) and essential morphosyntactic features (UPOS tags, Vibhakti markers). We find that syntactic structure (dependencies) is primarily encoded in the middle-to-upper-middle layers (layers 6-9), while lexical features peak slightly earlier. Although the general layer-wise trends are shared across models, significant variations in absolute probing performance reflect differences in model capacity, pre-training data, and language-specific characteristics. These findings shed light on how theory-specific grammatical information emerges implicitly within multilingual transformer representations trained largely on unstructured raw text.

Abstract

The relationship between tokenizer algorithm (e.g., Byte-Pair Encoding (BPE), Unigram), morphological alignment, tokenization quality (e.g., compression efficiency), and downstream performance remains largely unclear, particularly for languages with complex morphology. In this paper, we conduct a comprehensive evaluation of tokenizers using small-sized BERT models -- from pre-training through fine-tuning -- for Telugu (agglutinative), along with preliminary evaluation in Hindi (primarily fusional with some agglutination) and English (fusional). To evaluate morphological alignment of tokenizers in Telugu, we create a dataset containing gold morpheme segmentations of 600 derivational and 7000 inflectional word forms.
Our experiments reveal two key findings for Telugu. First, the choice of tokenizer algorithm is the most significant factor influencing performance, with Unigram-based tokenizers consistently outperforming BPE across most settings. Second, while better morphological alignment shows a moderate, positive correlation with performance on text classification and structure prediction tasks, its impact is secondary to the tokenizer algorithm. Notably, hybrid approaches that use morphological information for pre-segmentation significantly boost the performance of BPE, though not Unigram. Our results further showcase the need for comprehensive intrinsic evaluation metrics for tokenizers that could explain downstream performance trends consistently.

Abstract

Identification of hallucination spans in black-box language model generated text is essential for applications in the real world. A recent attempt at this direction is SemEval-2025 Task 3, Mu-SHROOM-a Multilingual Shared Task on Hallucinations and Related Observable Over-generation Errors. In this work, we present our solution to this problem, which capitalizes on the variability of stochastically-sampled responses in order to identify hallucinated spans. Our hypothesis is that if a language model is certain of a fact, its sampled responses will be uniform, while hallucinated facts will yield different and conflicting results. We measure this divergence through entropy-based analysis, allowing for accurate identification of hallucinated segments. Our method is not dependent on additional training and hence is cost-effective and adaptable. In addition, we conduct extensive hyperparameter tuning and perform error analysis, giving us crucial insights into model behavior.

Abstract

We present LayForge, a two-track lay sum-
mary generation system developed for the Bio-
LaySumm2025 shared task(Xiao et al., 2025)
. Task 1.1 addresses the lay summarization
using only the internal content of the article,
while Task 1.2 augments this process with do-
main knowledge such as biomedical definitions
and concept explanations. BioLaySumm em-
ploys a modular architecture that leverages
large language models (LLMs), a BioBERT-
based named entity recognizer (NER), and the
UMLS(Bodenreider, 2004) knowledge base to
create readable, informative, and faithful lay
summaries. Our system shows strong perfor-
mance on both tasks when evaluated on the
PLOS and elife subset(Goldsack et al., 2022),
particularly in readability and factuality metrics.
The architecture illustrates how modularity and
domain adaptation can be effectively combined
for accessible biomedical communication.

Abstract

In this study, we present our approach to SemEval Task 8: Question Answering over Tabular Data (Os'es Grijalba et al., 2025), where we develop a Large Language Models(LLM) based agent capable of answering questions over tabular data. Our agent leverages a set of custom defined tools incorporating structured table parsing and reasoning mechanisms to enhance semantic understanding of tabular data. Extending our methodology, we apply chain-of- thought prompting to further refine it's understanding of the task. Our findings suggest that LLM-based agents, when properly adapted, can significantly improve their table-based question answering capabilities.