Abstract

Presidential debates provide a recurring, institu-
tionally comparable record of how candidates prioritize public
issues during election campaigns. This paper presents a longi-
tudinal analysis of policy agendas in U.S. presidential and vice-
presidential debates from 1960 to 2024. We analyze 48 debate
transcripts using a hybrid framework that combines interpretable
keyword-based topic identification with large language model
(LLM)-assisted semantic topic extraction and sentence-level topic
clustering. The analysis identifies durable and shifting com-
ponents of debate agendas across six decades. Foreign policy
and security are especially prominent in the early Cold War
era, domestic economic governance remains consistently central
across the entire period, and healthcare, immigration, climate,
and institutional concerns become increasingly visible in later
decades. In addition to topic prominence, we examine agenda
diversity and transitions between dominant policy domains across
election cycles. The results show that presidential debate agendas
evolve gradually rather than abruptly, while becoming more
topically diverse in recent decades. These findings demonstrate
the value of LLM-assisted methods for historically grounded
analysis of political discourse and for tracing long-term changes
in policy emphasis in democratic elections.

Abstract

Building upon the critical importance of ethical considerations in research, we introduce a novel task of Ethical
Questionnaire Generation (EQG) for research papers. Ethical review has become an indispensable component of
the research process, helping identify potential risks, biases, and societal impacts that may arise from scientific work.
In this paper, we present EthiQuest, a comprehensive dataset comprising 3663 research papers paired with their
corresponding ethical questionnaires extracted from major conference proceedings. We explore various approaches
leveraging large language models (LLMs) to automatically generate context-aware ethical questionnaires, examining
the unique challenges of capturing domain-specific ethical concerns, ensuring comprehensive coverage of potential
issues, and maintaining question relevance and clarity. Our experiments demonstrate the effectiveness of fine-tuned
LLMs in generating pertinent ethical questions across diverse research domains. We provide detailed analysis
of question quality, coverage metrics, and practical insights for deploying such systems in real-world research review processes. The EQG dataset and code can be accessed at https://github.com/Ishank-Kapania/eqg/.

Abstract

Large Language Models often achieve strong performance by generating long intermediate chain-of-thought reasoning. However, it remains unclear when a model's final answer is actually determined during generation. If the answer is already fixed at an intermediate stage, subsequent reasoning tokens may constitute post-decision explanation, increasing inference cost and latency without improving correctness. We study the evolution of predicted answers over reasoning steps using forced answer completion, which elicits the model's intermediate predictions at partial reasoning prefixes. Focusing on Qwen3-4B and averaging results across all datasets considered, we find that predicted answers change in only 32% of queries. Moreover, once the final answer switch occurs, the model generates an average of 760 additional reasoning tokens per query, accounting for a substantial fraction of the total reasoning budget. Motivated by these findings, we investigate early stopping strategies that halt generation once the answer has stabilized. We show that simple heuristics, including probe-based stopping, can reduce reasoning token usage by 500 tokens per query while incurring only a 2% drop in accuracy. Together, our results indicate that a large portion of chain-of-thought generation is redundant and can be reduced with minimal impact on performance.

Abstract

Large language models (LLMs) exhibit failures on elementary symbolic tasks such as character counting in a word, despite excelling on complex benchmarks. Although this limitation has been noted, the internal reasons remain unclear. We use character counting (e.g., "How many p's are in apple?") as a minimal, controlled probe that isolates token-level reasoning from higher-level confounds. Using this setting, we uncover a consistent phenomenon across modern architectures, including LLaMA, Qwen, and Gemma: models often compute the correct answer internally yet fail to express it at the output layer.
Through mechanistic analysis combining probing classifiers, activation patching, logit lens analysis, and attention head tracing, we show that character-level information is encoded in early and mid-layer representations. However, this information is attenuated by a small set of components in later layers, especially the penultimate and final layer MLP. We identify these components as negative circuits: subnetworks that downweight correct signals in favor of higher-probability but incorrect outputs.
Our results lead to two contributions. First, we show that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but arise from structured interference within the model's computation graph. This explains why such errors persist and can worsen under scaling and instruction tuning. Second, we provide evidence that LLM forward passes implement a form of competitive decoding, in which correct and incorrect hypotheses coexist and are dynamically reweighted, with final outputs determined by suppression as much as by amplification.
These findings carry implications for interpretability and robustness: simple symbolic reasoning exposes weaknesses in modern LLMs, underscoring need for design strategies that ensure information is encoded and reliably used.

Abstract

We introduce the CAP (Confabulations from ACL Publications) dataset, a multilingual resource for studying
hallucinations in large language models (LLMs) within scientific text generation. CAP focuses on the scientific
domain, where hallucinations can distort factual knowledge, as they frequently do. In this domain, however, the
presence of specialized terminology, statistical reasoning, and context-dependent interpretations further exacerbates
these distortions, particularly given LLMs' lack of true comprehension, limited contextual understanding, and bias
toward surface-level generalization. CAP operates in a cross-lingual setting covering five high-resource languages
(English, French, Hindi, Italian, and Spanish) and four low-resource languages (Bengali, Gujarati, Malayalam, and
Telugu). The dataset comprises 900 curated scientific questions and over 7,000 LLM-generated answers from
16 publicly available models, provided as question-answer pairs along with token sequences and corresponding
logits. Each instance is annotated with a binary label indicating the presence of a scientific hallucination, denoted as
a factuality error, and a fluency label, capturing issues in the linguistic quality or naturalness of the text. CAP is
publicly released to facilitate advanced research on hallucination detection, multilingual evaluation of LLMs, and the
development of more reliable scientific NLP systems.

Abstract

Large Language Models (LLMs) excel on English reasoning tasks but falter on morphologically rich, low-resource languages such as Telugu, Tamil, and Kannada. We present TeluguEval, a human-curated reasoning benchmark created by translating GSM8K (math), Winogrande (commonsense), ARC (science), CaseHOLD (law), and Hendrycks Ethics into Telugu. We evaluate eight models spanning global (Llama-3.1-8B, Llama-2-7B, Qwen-8B, Gemma-7B, Gemini-2.0) and regional (Telugu-Llama2-7B, Indic-Gemma-7B, Sarvam-m-24B) systems. While extremely strong models such as Gemini and Sarvam-m largely retain performance in Telugu, most English-centric models suffer severe accuracy drops, often exceeding 30 to 40 points, particularly on mathematical and scientific reasoning. We further observe systematic failure modes including script sensitivity, option-selection bias, repetition loops, and unintended code-switching. Our results demonstrate that surface-level Telugu fluency does not imply robust reasoning capability, underscoring the need for Telugu-specific data, tokenization, and pretraining. TeluguEval provides a standardized testbed to drive progress on reasoning in low-resource Indian languages.

Abstract

Generative artificial intelligence (AI), especially large language models (LLMs), is increasingly
deployed in domains such as recruitment, content creation, and education. While these systems
accelerate productivity, they also risk reproducing and amplifying societal biases (Ahuchogu et
al., 2025). This project addresses the urgent challenge of identifying, quantifying, and mitigating
gender bias in text-generative AI outputs, with a focus on job narratives. Building on my
independent study of 11,000+ AI-generated job narratives, which we generated using Gemini AI,
we introduce a bias quantification framework using mean bias, mean absolute bias, sentiment
skew (via TextBlob), and distributional measures (via Kullback-Leibler divergence and related
distances). Preliminary results show measurable gendered patterns across generated narratives,
validating the hypothesis of proposed gender bias in LLM.
The proposed work extends this foundation in three directions: expanding bias quantification
using probabilistic distribution distances (Devisetti, 2024)(Chung et al., 1989), evaluating
prompt-construction bias and multi-model comparisons across GPT-3, GPT-4, Gemini, and
open-source LLMs (Blodgett et al., 2020), and integrating interpretable embedding methods
(e.g., SPINE)(Subramanian et al., 2017) for transparency in downstream debiasing.
The expected contribution is both theoretical and practical: a robust bias quantification pipeline
grounded in probability theory, and actionable strategies to mitigate bias in LLM-generated
recruitment texts(Ferrara, 2024). Beyond recruitment, the proposed methodology aims to serve
as a standard for bias evaluation in generative AI applications more broadly.

Abstract

Transformer-based models have revolutionized the field of natural language processing. To understand why they perform so well and to assess their reliability, several studies have focused on questions such as: Which linguistic properties are encoded by these models, and to what extent? How robust are these models in encoding linguistic properties when faced with perturbations in the input text? However, these studies have mainly focused on BERT and the English language. In this paper, we investigate similar questions regarding encoding capability and robustness for 8 linguistic properties across 13 different perturbations in 6 Indic languages, using 9 multilingual Transformer models (7 universal and 2 Indic-specific). To conduct this study, we introduce a novel multilingual benchmark dataset, IndicSentEval, containing approximately ~47K sentences. Our probing analysis of surface, syntactic, and semantic properties reveals that, while almost all multilingual models demonstrate consistent encoding performance for English, surprisingly, they show mixed results for Indic languages. As expected, Indic-specific multilingual models capture linguistic properties in Indic languages better than universal models. Intriguingly, universal models broadly exhibit better robustness compared to Indic-specific models, particularly under perturbations such as dropping both nouns and verbs, dropping only verbs, or keeping only nouns. Overall, this study provides valuable insights into probing and perturbation-specific strengths and weaknesses of popular multilingual Transformer-based models for different Indic languages.

Abstract

Over 46 million people worldwide suffer from
dysarthria--a motor speech disorder caused by
neurological conditions like stroke or Parkin
son's disease--making their speech slurred, un
intelligible, and emotionally distorted. This
severely affects communication, quality of life,
and social inclusion.
Wepresent the first emotion-aware framework
for dysarthric speech reconstruction, where the
speaker's emotion is detected from audio and
used to guide large language models in recover
ing intelligible, emotionally faithful sentences.
To evaluate this, we introduce a novel
metric--Multimodal
Communication
Dysarthria Score (MCDS)--which holisti
cally measures both linguistic and emotional
accuracy. Our results show strong improve
ments over traditional baselines, offering a
breakthrough toward emotionally intelligent
assistive speech systems that prioritize both
understanding and empathy

Abstract

Although speech language models are expected
to align well with brain language processing
during speech comprehension, recent studies
have found that they fail to capture brain
relevant semantics beyond low-level features.
Surprisingly, text-based language models ex
hibit stronger alignment with brain language
regions, as they better capture brain-relevant
semantics. However, no prior work has exam
ined the alignment effectiveness of text/speech
representations from multimodal models. This
raises several key questions: Can speech em
beddings from such multimodal models capture
brain-relevant semantics through cross-modal
interactions? Which modality can take advan
tage of this synergistic multimodal understand
ing to improve alignment with brain language
processing? Can text/speech representations
from such multimodal models outperform uni
modal models? To address these questions,
we systematically analyze multiple multimodal
models, extracting both text- and speech-based
representations to assess their alignment with
MEGbrain recordings during naturalistic story
listening. We find that text embeddings from
both multimodal and unimodal models signif
icantly outperform speech embeddings from
these models. Specifically, multimodal text
embeddings exhibit a peak around 200 ms, sug
gesting that they benefit from speech embed
dings, with heightened activity during this time
period. However, speech embeddings from
these multimodal models still show a similar
alignment compared to their unimodal counter
parts, suggesting that they do not gain meaning
ful semantic benefits over text-based represen
tations. These results highlight an asymmetry
in cross-modal knowledge transfer, where the
text modality benefits more from speech infor
mation, but not vice versa. We make the code
publicly available.