Abstract

LLM-as-a-Judge frameworks have emerged as scalable alternatives
to human evaluation, with recent work reporting 70-80% agreement
with human evaluators. However, overall agreement can mask systematic
attribute-level misalignment--a critical concern for educational
validity. We investigate this phenomenon in automated essay evaluation
for English language learners, where the assessment quality depends on
the pedagogically appropriate weighting of criteria such as cohesion, syntax,
and conventions. Through systematic evaluation of six LLMs on
1100 ELL essays, we find that even when most models achieve a reasonable
overall quality agreement (ρ = 0.6-0.74), they exhibit severe
attribution biases. Direct prompting of LLaMA-3.1-8B reveals 18.8%
syntax underweighting and 18.7% grammar overweighting (11% mean
divergence), systematically showing different associations with linguistic
complexity over surface correctness. To address these misalignments, we
propose knowledge distillation based on LLM reasoning traces with explicit
interpretability mechanisms: attribution weighting and consistency
regularization. Our best interpretability-optimized model (AD) outperforms
direct LLM prompting in both prediction accuracy (Spearman ρ =
0.763 vs 0.51, +50% improvement) and attribution alignment (3.2% vs
11.0% mean divergence, -71% reduction), while being more computationally
efficient. The findings establish knowledge distillation as a superior
method to direct prompting for educational assessment, enabling computationally
efficient, accurate, and evaluation-rubric-aligned automated
feedback for English language learners at scale.

Abstract

have emerged as a scalable alternative to expensive
human evaluations. Although, LLM
judges demonstrate 70-80% agreement with
human evaluators, their robustness under semantically
equivalent prompt variations remain
underexplored. Through systematic evaluation
of 8 models across 4 NLG tasks using 10 semantically
equivalent paraphrases per prompt
(≈ 115000 evaluations), we identify a critical
accuracy-robustness gap: attribute verifiability
affects the robustness more than model choice,
with factually verifiable attributes achieving
0.71 accuracy versus 0.19 for subjective attributes.
Our investigations discover three key
insights: 1) Task structure characteristics influence
the robustness and in turn accuracy, 2)
Attribute verifiability as the strongest predictor factually
verifiable attribute achieve 0.71 accuracy
versus 0.19 for subjective attributes,
3) No single winning model-smallest model
(LLaMA-3.1-8B) exhibits second-best performance,
while the strongest model (LLaMA-
4) from the same family significantly lag behind,
thus demonstrating that general capability
improvements do not necessarily result in
evaluation robustness. With these findings, we
propose a diagnostic framework grounded in
attribute verifiability that enables principled decisions
about evaluation automation. Our work
establishes new standards for assessing LLM
judge reliability beyond simple accuracy metrics.

Abstract

Discourse-level translation remains a challenge for machine translation (MT) systems, particularly for translation from Indian languages to English. This challenge is amplified in Indian languages due to rich morphology and discourse complexity. Existing evaluation benchmarks prioritize sentence-level translation quality and fail to capture important discourse phenomena such as pronoun resolution and lexical cohesion. To address this gap, we introduce IndicDISCO-MT, a parallel benchmark dataset covering translations from eight Indian languages, namely: Bengali, Gujarati, Hindi, Marathi, Kannada, Tamil, Telugu and Urdu to English. On top of this corpus, we introduce DiscoAlign, a human-annotated source-to-target alignment pairs for discourse evaluation. We further propose two evaluation benchmarks, ProAlign and LexiAlign, to evaluate ability of large language models (LLMs) and MT systems to handle personal pronouns and lexical cohesion. Our evaluation of recent LLM and MT systems shows that although models achieve high translation quality, they still struggle to accurately preserve discourse-level phenomena. The proposed benchmarks provide a systematic framework for evaluating discourse-aware translation and can facilitate the development of more coherent and contextually consistent translations. The dataset is made publicly available.

Abstract

Architecture Knowledge Management (AKM) is crucial for maintaining current and comprehensive software Architecture Knowledge (AK) in a software project. However AKM is often a laborious process and is not adopted by developers and architects. While LLMs present an opportunity for automation, a naive, single-prompt approach is often ineffective, constrained by context limits and an inability to grasp the distributed nature of architectural knowledge. To address these limitations, we propose an Agentic approach for AKM, AgenticAKM, where the complex problem of architecture recovery and documentation is decomposed into manageable sub-tasks. Specialized agents for architecture Extraction, Retrieval, Generation, and Validation collaborate in a structured workflow to generate AK. To validate we made an initial instantiation of our approach to generate Architecture Decision Records (ADRs) from code repositories. We validated our approach through a user study with 29 repositories. The results demonstrate that our agentic approach generates better ADRs, and is a promising and practical approach for automating AKM.

Abstract

Large language models are going silent. As mathematical reasoning evolves
from explicit Chain-of-Thought to efficient "latent reasoning," models
using methods like Chain of Continuous Thought (CoConut) no longer show
their work; they "think" through continuous, inscrutable hidden states.

For education, this creates a critical transparency crisis: how can we
trust an AI tutor that solves problems without revealing its reasoning
process? When should it intervene if we cannot detect its uncertainty
or errors?

We break open this black box by applying Sparse Autoencoders (SAEs) to
interpret CoConut's internal reasoning states during step-by-step problem
solving on GSM8k. Through systematic analysis of 7,914 reasoning steps
and manual annotation of problems across 9 mathematical skills, we
discover something remarkable: buried within these inscrutable activations
are interpretable features that reliably detect specific reasoning patterns
(F1 = 0.542, 135% better than random).

When we surgically ablate these features, performance collapses. Removing
the top feature causes a 44.6% drop, with some skills degrading by 67%,
proving these aren't spurious correlations but causally important mechanisms.

Most strikingly, we reveal that features detecting uncertainty and logical
contradictions are nearly perfectly correlated (r = 0.988, p < 0.0001):
the model maintains a unified internal "reasoning quality monitor."

These findings reveal that latent reasoning, despite its opacity, contains
structured representations that enable skill detection and error monitoring,
providing a foundation for more transparent mathematical reasoning systems.

Keywords: Sparse Autoencoders · Latent Reasoning · Mechanistic Interpretability
· AI Tutoring · Error Detection

Abstract

A Message Sequence Chart (MSC) is a visually comprehensible knowledge representation used for showing events of a narrative in their correct temporal order. However, to enable its use as a spatio-temporal knowledge representation, it needs to be enhanced with spatial knowledge present in the narrative. In this work, we aim at enriching MSCs with spatial knowledge, using their condition construct, to enable them for downstream spatio-temporal applications. We propose two Large Language Model (LLM) based approaches for MSC construction, one based on few-shot prompting and another involving fine-tuning. Further, we develop a few-shot prompting approach to enable the spatial enrichment of the constructed MSCs. We report on performance of the proposed approaches and also demonstrate the utility of the enrichment in an image-based visualization application.

Abstract

A Message Sequence Chart (MSC) is a visually comprehensible knowledge representation used for showing events of a narrative in their correct temporal order. However, to enable its use as a spatio-temporal knowledge representation, it needs to be enhanced with spatial knowledge present in the narrative. In this work, we aim at enriching MSCs with spatial knowledge about locations of the involved entities, using their condition construct, to enable them for downstream spatio-temporal applications. We propose two Large Language Model (LLM) based approaches for MSC construction, one based on few-shot in-context learning and another involving fine-tuning. Further, we propose a new task of motion reasoning, where an entity's location is reasoned based on the last event it was part of and whether the event involved any motion causing change of the entity's location. Similar as earlier, we develop a few-shot in-context learning approach and an LLM fine-tuning in-context learning to solve the motion reasoning task and enable the spatial enrichment of the constructed MSCs. We report performance of the proposed approaches and also demonstrate the utility of the enrichment in two applications on image-based visualization and narrative gap finding.

Abstract

Hallucinations in large language models(LLMs) have recently become a significantproblem. A recent effort in this directionis a shared task at Semeval 2024 Task 6,SHROOM, a Shared-task on Hallucinationsand Related Observable Overgeneration Mis-takes. This paper describes our winning so-lution ranked 1st and 2nd in the 2 sub-tasksof model agnostic and model aware tracks re-spectively. We propose a meta-regressor basedensemble of LLMs based on a random forestalgorithm that achieves the highest scores onthe leader board. We also experiment with var-ious transformer based models and black boxmethods like ChatGPT, Vectara, and others. Inaddition, we perform an error analysis com-paring ChatGPT against our best model whichshows the limitations of the former

Abstract

Wikipedia is a highly essential platform because of its informative, dynamic, and easily accessible nature. To identify topics/titles warranting their own Wikipedia article, editors of Wikipedia defined "Notability" guidelines. So far notability is enforced by humans, which makes scalability an issue. There has been no significant work on Notability determination for titles with complex category dependencies. We design a mechanism to identify such titles. We construct a dataset with 9k such titles and propose a category-agnostic approach utilizing Graph neural networks, for their notability determination. Our system outperforms machine learning-based, transformer-based classifiers and entity salience methods. It provides a scalable alternative for notability detection.

Abstract

Curriculum learning has been used to improve the quality of text generation systems by ordering the training samples according to a particular schedule in various tasks. In the context of data-to-text generation (DTG), previous studies used various difficulty criteria to order the training samples for monolingual DTG. These criteria, however, do not generalize to the crosslingual variant of the problem and do not account for noisy data. We explore multiple criteria that can be used for improving the performance of cross-lingual DTG systems with noisy data using two curriculum schedules. Using the alignment score criterion for ordering samples and an annealing schedule to train the model, we show increase in BLEU score by up to 4 points, and improvements in faithfulness and coverage of generations by 5-15% on average across 11 Indian languages and English in 2 separate datasets. We make code and data publicly available