Abstract

Discourse-level translation remains a challenge for machine translation (MT) systems, particularly for translation from Indian languages to English. This challenge is amplified in Indian languages due to rich morphology and discourse complexity. Existing evaluation benchmarks prioritize sentence-level translation quality and fail to capture important discourse phenomena such as pronoun resolution and lexical cohesion. To address this gap, we introduce IndicDISCO-MT, a parallel benchmark dataset covering translations from eight Indian languages, namely: Bengali, Gujarati, Hindi, Marathi, Kannada, Tamil, Telugu and Urdu to English. On top of this corpus, we introduce DiscoAlign, a human-annotated source-to-target alignment pairs for discourse evaluation. We further propose two evaluation benchmarks, ProAlign and LexiAlign, to evaluate ability of large language models (LLMs) and MT systems to handle personal pronouns and lexical cohesion. Our evaluation of recent LLM and MT systems shows that although models achieve high translation quality, they still struggle to accurately preserve discourse-level phenomena. The proposed benchmarks provide a systematic framework for evaluating discourse-aware translation and can facilitate the development of more coherent and contextually consistent translations. The dataset is made publicly available.

Abstract

Product reviews on e-commerce platforms are a critical form of user-generated content that influence consumer decisions. However, these reviews are predominantly in English, creating a significant accessibility barrier for users who are not fluent in English. When translating into major Indian languages using the current models, the outputs often fail to capture domain-specific features and colloquial style, resulting in stylistically unnatural texts. To address this gap, we introduce textbf{STAR-IL}, a human-annotated, multilingual, parallel corpus for style-aware translation of product reviews. We evaluate the performance of several state-of-the-art models on our dataset for the task of product review translation. Our experiments show that models fine-tuned on STAR-IL achieve significant average performance gain of textbf{5.77} points in BLEU and textbf{3.78} points in COMET, when compared to their baselines, across all languages. Our dataset provides a valuable benchmark for future research in style-aware product review translation. The STAR-IL dataset is publicly available at https://github.com/ltrc/STAR-IL-Corpus

Abstract

We study Gemma-2-2B on a controlled role-gated retrieval task where a prepositional gate (to or from) selects which of two entities is correct. On 60 single-token name pairs the model attains 100% accuracy with a mean flip magnitude (~3.5) (sum of per-condition correctness margins). Using causal tracing, we identify a Query-Gated Courier circuit with three stages: (1) a gate token (from/to) writes a role feature at the answer; (2) this feature perturbs late-layer courier queries, shifting their (q · k) preference; (3) couriers attend to the correct name and inject it via OV, raising its logit. Gate-residual swaps flip predictions, and a compact nine-head keep set reproduces the behavior with high fidelity. The circuit gives a potential algorithm for role tracking and aligns with the Paninian Karaka analysis, mapping to to sampradana and from to apadana.

Abstract

Large multilingual transformers such as XLM-RoBERTa achieve impressive performance on diverse NLP benchmarks, but understanding how they internally encode grammatical information remains challenging. This study investigates the encoding of syntactic and morphological information derived from the Paninian grammatical framework--specifically designed for morphologically rich Indian languages--across model layers. Using diagnostic probing, we analyze the hidden representations of frozen XLM-RoBERTa-base, mBERT, and IndicBERT models across seven Indian languages (Hindi, Kannada, Malayalam, Marathi, Telugu, Urdu, Bengali). Probes are trained to predict Paninian dependency relations (by edge probing) and essential morphosyntactic features (UPOS tags, Vibhakti markers). We find that syntactic structure (dependencies) is primarily encoded in the middle-to-upper-middle layers (layers 6-9), while lexical features peak slightly earlier. Although the general layer-wise trends are shared across models, significant variations in absolute probing performance reflect differences in model capacity, pre-training data, and language-specific characteristics. These findings shed light on how theory-specific grammatical information emerges implicitly within multilingual transformer representations trained largely on unstructured raw text.

Abstract

The relationship between tokenizer algorithm (e.g., Byte-Pair Encoding (BPE), Unigram), morphological alignment, tokenization quality (e.g., compression efficiency), and downstream performance remains largely unclear, particularly for languages with complex morphology. In this paper, we conduct a comprehensive evaluation of tokenizers using small-sized BERT models -- from pre-training through fine-tuning -- for Telugu (agglutinative), along with preliminary evaluation in Hindi (primarily fusional with some agglutination) and English (fusional). To evaluate morphological alignment of tokenizers in Telugu, we create a dataset containing gold morpheme segmentations of 600 derivational and 7000 inflectional word forms.
Our experiments reveal two key findings for Telugu. First, the choice of tokenizer algorithm is the most significant factor influencing performance, with Unigram-based tokenizers consistently outperforming BPE across most settings. Second, while better morphological alignment shows a moderate, positive correlation with performance on text classification and structure prediction tasks, its impact is secondary to the tokenizer algorithm. Notably, hybrid approaches that use morphological information for pre-segmentation significantly boost the performance of BPE, though not Unigram. Our results further showcase the need for comprehensive intrinsic evaluation metrics for tokenizers that could explain downstream performance trends consistently.

Abstract

This study addresses the critical challenge of
data scarcity in machine translation for Indian
languages, particularly given their morpholog-
ical complexity and limited parallel data. We
investigate an effective strategy to maximize
the utility of existing data by generating nega-
tive samples from positive training instances us-
ing a progressive perturbation approach. This
is used to align the model with preferential
data using Kahneman-Tversky Optimization
(KTO). Comparing it against traditional Su-
pervised Fine-Tuning (SFT), we demonstrate
how generating negative samples and leverag-
ing KTO enhances data efficiency. By creat-
ing rejected samples through progressively per-
turbed translations from the available dataset,
we fine-tune the Llama 3.1 Instruct 8B model
using QLoRA across 16 language directions, in-
cluding English, Hindi, Bangla, Tamil, Telugu,
and Santali. Our results show that KTO-based
preference alignment with progressive pertur-
bation consistently outperforms SFT, achieving
significant gains in translation quality with an
average BLEU increase of 1.84 to 2.47 and
CHRF increase of 2.85 to 4.01 compared to
SFT for selected languages, while using the
same positive training samples and under simi-
lar computational constraints. This highlights
the potential of our negative sample genera-
tion strategy within KTO, especially in low-
resource scenarios.

Abstract

With the primary focus on evaluating the ef- fectiveness of large language models for au- tomatic reference-less translation assessment, this work presents our experiments on mimick- ing human direct assessment to evaluate the quality of translations in English and Indian languages. We constructed a translation evalua- tion task where we performed zero-shot learn- ing, in-context example-driven learning, and fine-tuning of large language models to pro- vide a score out of 100, where 100 represents a perfect translation and 1 represents a poor translation. We compared the performance of our trained systems with existing methods such as COMET, BERT-Scorer, and LABSE, and found that the LLM-based evaluator (LLaMA- 2-13B) achieves a comparable or higher overall correlation with human judgments for the con- sidered Indian language pairs (Refer figure 1).

Abstract

The objective of the MTIL track in FIRE 2023 was to encourage the development of Indian Language to Indian Language (IL-IL) Neural Machine Translation models. The languages covered in the track included Hindi, Gujarati, Kannada, Odia, Punjabi, Urdu, Telugu, Kashmiri, and Sindhi. The track consists of two tasks: (i) a General Translation Task and (ii) a Domain specific Translation Task with Governance and Healthcare being the chosen domains. For the listed languages, we proposed 12 diverse language directions for the general domain translation task and 8 each for healthcare and governance domains. Participants were encouraged to submit models for one or more language pairs. We witnessed the creation of 34 distinct models spanning various language pairs and domains. Model assessments were conducted using five evaluation metrics: BLEU, CHRF, CHRF++, TER, and COMET. The submitted model outputs were ranked based on the CHRF score.

Abstract

How can we effectively estimate the quality of translated texts in the medical field, where back-translation is
usually available and/or recommended for sensitive documents. This paper proposes a novel metric, GATE1
,
for translation quality estimation task, leveraging the Resource Description Framework (RDF) to encode both
semantic and syntactical information of the original and back-translated sentences into RDF graphs. The distance
between these graphs is measured to get the semantic similarity score to assess the quality of the translation.
Unlike traditional metrics like BLEU and METEOR, our approach is reference-less, capturing both semantic and
syntactical information for a comprehensive assessment of translation quality. Our results correlate better with
human judgment, giving a better Pearson correlation (0.357) as compared to BLEU (0.200), thereby showing ~70%
improvement over BLEU. Our research shows that, in the field of translation evaluation, existing resources like
back-translation and RDF could be useful.

Abstract

How can we effectively estimate the quality of translated texts in the medical field, where back-translation is
usually available and/or recommended for sensitive documents. This paper proposes a novel metric, GATE 1 ,
for translation quality estimation task, leveraging the Resource Description Framework (RDF) to encode both
semantic and syntactical information of the original and back-translated sentences into RDF graphs. The distance
between these graphs is measured to get the semantic similarity score to assess the quality of the translation.
Unlike traditional metrics like BLEU and METEOR, our approach is reference-less, capturing both semantic and
syntactical information for a comprehensive assessment of translation quality. Our results correlate better with
human judgment, giving a better Pearson correlation (0.357) as compared to BLEU (0.200), thereby showing ~70%
improvement over BLEU. Our research shows that, in the field of translation evaluation, existing resources like
back-translation and RDF could be useful.