Abstract

Architecture Knowledge Management (AKM) is crucial for maintaining current and comprehensive software Architecture Knowledge (AK) in a software project. However AKM is often a laborious process and is not adopted by developers and architects. While LLMs present an opportunity for automation, a naive, single-prompt approach is often ineffective, constrained by context limits and an inability to grasp the distributed nature of architectural knowledge. To address these limitations, we propose an Agentic approach for AKM, AgenticAKM, where the complex problem of architecture recovery and documentation is decomposed into manageable sub-tasks. Specialized agents for architecture Extraction, Retrieval, Generation, and Validation collaborate in a structured workflow to generate AK. To validate we made an initial instantiation of our approach to generate Architecture Decision Records (ADRs) from code repositories. We validated our approach through a user study with 29 repositories. The results demonstrate that our agentic approach generates better ADRs, and is a promising and practical approach for automating AKM.

Abstract

Large language models are going silent. As mathematical reasoning evolves
from explicit Chain-of-Thought to efficient "latent reasoning," models
using methods like Chain of Continuous Thought (CoConut) no longer show
their work; they "think" through continuous, inscrutable hidden states.

For education, this creates a critical transparency crisis: how can we
trust an AI tutor that solves problems without revealing its reasoning
process? When should it intervene if we cannot detect its uncertainty
or errors?

We break open this black box by applying Sparse Autoencoders (SAEs) to
interpret CoConut's internal reasoning states during step-by-step problem
solving on GSM8k. Through systematic analysis of 7,914 reasoning steps
and manual annotation of problems across 9 mathematical skills, we
discover something remarkable: buried within these inscrutable activations
are interpretable features that reliably detect specific reasoning patterns
(F1 = 0.542, 135% better than random).

When we surgically ablate these features, performance collapses. Removing
the top feature causes a 44.6% drop, with some skills degrading by 67%,
proving these aren't spurious correlations but causally important mechanisms.

Most strikingly, we reveal that features detecting uncertainty and logical
contradictions are nearly perfectly correlated (r = 0.988, p < 0.0001):
the model maintains a unified internal "reasoning quality monitor."

These findings reveal that latent reasoning, despite its opacity, contains
structured representations that enable skill detection and error monitoring,
providing a foundation for more transparent mathematical reasoning systems.

Keywords: Sparse Autoencoders · Latent Reasoning · Mechanistic Interpretability
· AI Tutoring · Error Detection

Abstract

A Message Sequence Chart (MSC) is a visually comprehensible knowledge representation used for showing events of a narrative in their correct temporal order. However, to enable its use as a spatio-temporal knowledge representation, it needs to be enhanced with spatial knowledge present in the narrative. In this work, we aim at enriching MSCs with spatial knowledge, using their condition construct, to enable them for downstream spatio-temporal applications. We propose two Large Language Model (LLM) based approaches for MSC construction, one based on few-shot prompting and another involving fine-tuning. Further, we develop a few-shot prompting approach to enable the spatial enrichment of the constructed MSCs. We report on performance of the proposed approaches and also demonstrate the utility of the enrichment in an image-based visualization application.

Abstract

A Message Sequence Chart (MSC) is a visually comprehensible knowledge representation used for showing events of a narrative in their correct temporal order. However, to enable its use as a spatio-temporal knowledge representation, it needs to be enhanced with spatial knowledge present in the narrative. In this work, we aim at enriching MSCs with spatial knowledge about locations of the involved entities, using their condition construct, to enable them for downstream spatio-temporal applications. We propose two Large Language Model (LLM) based approaches for MSC construction, one based on few-shot in-context learning and another involving fine-tuning. Further, we propose a new task of motion reasoning, where an entity's location is reasoned based on the last event it was part of and whether the event involved any motion causing change of the entity's location. Similar as earlier, we develop a few-shot in-context learning approach and an LLM fine-tuning in-context learning to solve the motion reasoning task and enable the spatial enrichment of the constructed MSCs. We report performance of the proposed approaches and also demonstrate the utility of the enrichment in two applications on image-based visualization and narrative gap finding.

Abstract

Hallucinations in large language models(LLMs) have recently become a significantproblem. A recent effort in this directionis a shared task at Semeval 2024 Task 6,SHROOM, a Shared-task on Hallucinationsand Related Observable Overgeneration Mis-takes. This paper describes our winning so-lution ranked 1st and 2nd in the 2 sub-tasksof model agnostic and model aware tracks re-spectively. We propose a meta-regressor basedensemble of LLMs based on a random forestalgorithm that achieves the highest scores onthe leader board. We also experiment with var-ious transformer based models and black boxmethods like ChatGPT, Vectara, and others. Inaddition, we perform an error analysis com-paring ChatGPT against our best model whichshows the limitations of the former

Abstract

Wikipedia is a highly essential platform because of its informative, dynamic, and easily accessible nature. To identify topics/titles warranting their own Wikipedia article, editors of Wikipedia defined "Notability" guidelines. So far notability is enforced by humans, which makes scalability an issue. There has been no significant work on Notability determination for titles with complex category dependencies. We design a mechanism to identify such titles. We construct a dataset with 9k such titles and propose a category-agnostic approach utilizing Graph neural networks, for their notability determination. Our system outperforms machine learning-based, transformer-based classifiers and entity salience methods. It provides a scalable alternative for notability detection.

Abstract

Curriculum learning has been used to improve the quality of text generation systems by ordering the training samples according to a particular schedule in various tasks. In the context of data-to-text generation (DTG), previous studies used various difficulty criteria to order the training samples for monolingual DTG. These criteria, however, do not generalize to the crosslingual variant of the problem and do not account for noisy data. We explore multiple criteria that can be used for improving the performance of cross-lingual DTG systems with noisy data using two curriculum schedules. Using the alignment score criterion for ordering samples and an annealing schedule to train the model, we show increase in BLEU score by up to 4 points, and improvements in faithfulness and coverage of generations by 5-15% on average across 11 Indian languages and English in 2 separate datasets. We make code and data publicly available

Abstract

A visual question answering (VQA) system for electrical circuit images could be useful as a quiz generator, design and verification assistant or an electrical diagnosis tool. Although there exists a vast literature on VQA, to the best of our knowledge, there is no existing work on VQA for electrical circuit images. To this end, we curate a new dataset, CircuitVQA, of 115K+ questions on 5725 electrical images with
70 circuit symbols. The dataset contains schematic as well as hand-drawn images. The questions span various categories like counting, value, junction and position based questions. To be effective, models must demonstrate skills like object detection, text recognition, spatial understanding, question intent understanding and answer generation. We experiment with multiple foundational visio-linguistic models for this task and find that a finetuned BLIP model with component descriptions as additional input provides best results. We make the code and dataset publicly available

Abstract

To employ Pre-trained Language Models (PLMs) as knowledge containers in niche domains it is important to gauge the knowledge of these PLMs about facts in these domains. It is also an important pre-requisite to know how much enrichment effort is required to make them better. As part of this work, we aim to gauge and enrich small PLMs for knowledge of world geography. Firstly, we develop a moderately sized dataset of masked sentences covering 24 different fact types about world geography to estimate knowledge of PLMs on these facts. We hypothesize that for this niche domain, smaller PLMs may not be well equipped. Secondly, we enrich PLMs with this knowledge through fine-tuning and check if the knowledge in the dataset is infused sufficiently. We further hypothesize that linguistic variability in the manual templates used to embed the knowledge in masked sentences does not affect the knowledge infusion. Finally, we demonstrate the application of PLMs to tourism blog search and Wikidata KB augmentation. In both applications, we aim at showing the effectiveness of using PLMs to achieve competitive performance.

Abstract

Traditional evaluation metrics like BLEU and ROUGE fall short when capturing the nuanced qualities of generated text, particularly when there is no single ground truth. In this paper, we explore the potential of Large Language Models (LLMs), specifically Google Gemini 1, to serve as automatic evaluators for non-standardized metrics in summarization and dialog-based tasks. We conduct experiments across multiple prompting strategies to examine how LLMs fare as quality evaluators when compared with human judgments on the SummEval and USR datasets, asking the model to generate both a score as well as a justification for the score. Furthermore, we explore the robustness of the LLM evaluator by using perturbed inputs. Our findings suggest that while LLMs show promise, their alignment with human evaluators is limited, they are not robust against perturbations and significant improvements are required for their standalone use as reliable evaluators for subjective metrics.