Abstract

We introduce the CAP (Confabulations from ACL Publications) dataset, a multilingual resource for studying
hallucinations in large language models (LLMs) within scientific text generation. CAP focuses on the scientific
domain, where hallucinations can distort factual knowledge, as they frequently do. In this domain, however, the
presence of specialized terminology, statistical reasoning, and context-dependent interpretations further exacerbates
these distortions, particularly given LLMs' lack of true comprehension, limited contextual understanding, and bias
toward surface-level generalization. CAP operates in a cross-lingual setting covering five high-resource languages
(English, French, Hindi, Italian, and Spanish) and four low-resource languages (Bengali, Gujarati, Malayalam, and
Telugu). The dataset comprises 900 curated scientific questions and over 7,000 LLM-generated answers from
16 publicly available models, provided as question-answer pairs along with token sequences and corresponding
logits. Each instance is annotated with a binary label indicating the presence of a scientific hallucination, denoted as
a factuality error, and a fluency label, capturing issues in the linguistic quality or naturalness of the text. CAP is
publicly released to facilitate advanced research on hallucination detection, multilingual evaluation of LLMs, and the
development of more reliable scientific NLP systems.

Abstract

Large Language Models (LLMs) excel on English reasoning tasks but falter on morphologically rich, low-resource languages such as Telugu, Tamil, and Kannada. We present TeluguEval, a human-curated reasoning benchmark created by translating GSM8K (math), Winogrande (commonsense), ARC (science), CaseHOLD (law), and Hendrycks Ethics into Telugu. We evaluate eight models spanning global (Llama-3.1-8B, Llama-2-7B, Qwen-8B, Gemma-7B, Gemini-2.0) and regional (Telugu-Llama2-7B, Indic-Gemma-7B, Sarvam-m-24B) systems. While extremely strong models such as Gemini and Sarvam-m largely retain performance in Telugu, most English-centric models suffer severe accuracy drops, often exceeding 30 to 40 points, particularly on mathematical and scientific reasoning. We further observe systematic failure modes including script sensitivity, option-selection bias, repetition loops, and unintended code-switching. Our results demonstrate that surface-level Telugu fluency does not imply robust reasoning capability, underscoring the need for Telugu-specific data, tokenization, and pretraining. TeluguEval provides a standardized testbed to drive progress on reasoning in low-resource Indian languages.

Abstract

Generative artificial intelligence (AI), especially large language models (LLMs), is increasingly
deployed in domains such as recruitment, content creation, and education. While these systems
accelerate productivity, they also risk reproducing and amplifying societal biases (Ahuchogu et
al., 2025). This project addresses the urgent challenge of identifying, quantifying, and mitigating
gender bias in text-generative AI outputs, with a focus on job narratives. Building on my
independent study of 11,000+ AI-generated job narratives, which we generated using Gemini AI,
we introduce a bias quantification framework using mean bias, mean absolute bias, sentiment
skew (via TextBlob), and distributional measures (via Kullback-Leibler divergence and related
distances). Preliminary results show measurable gendered patterns across generated narratives,
validating the hypothesis of proposed gender bias in LLM.
The proposed work extends this foundation in three directions: expanding bias quantification
using probabilistic distribution distances (Devisetti, 2024)(Chung et al., 1989), evaluating
prompt-construction bias and multi-model comparisons across GPT-3, GPT-4, Gemini, and
open-source LLMs (Blodgett et al., 2020), and integrating interpretable embedding methods
(e.g., SPINE)(Subramanian et al., 2017) for transparency in downstream debiasing.
The expected contribution is both theoretical and practical: a robust bias quantification pipeline
grounded in probability theory, and actionable strategies to mitigate bias in LLM-generated
recruitment texts(Ferrara, 2024). Beyond recruitment, the proposed methodology aims to serve
as a standard for bias evaluation in generative AI applications more broadly.

Abstract

Transformer-based models have revolutionized the field of natural language processing. To understand why they perform so well and to assess their reliability, several studies have focused on questions such as: Which linguistic properties are encoded by these models, and to what extent? How robust are these models in encoding linguistic properties when faced with perturbations in the input text? However, these studies have mainly focused on BERT and the English language. In this paper, we investigate similar questions regarding encoding capability and robustness for 8 linguistic properties across 13 different perturbations in 6 Indic languages, using 9 multilingual Transformer models (7 universal and 2 Indic-specific). To conduct this study, we introduce a novel multilingual benchmark dataset, IndicSentEval, containing approximately ~47K sentences. Our probing analysis of surface, syntactic, and semantic properties reveals that, while almost all multilingual models demonstrate consistent encoding performance for English, surprisingly, they show mixed results for Indic languages. As expected, Indic-specific multilingual models capture linguistic properties in Indic languages better than universal models. Intriguingly, universal models broadly exhibit better robustness compared to Indic-specific models, particularly under perturbations such as dropping both nouns and verbs, dropping only verbs, or keeping only nouns. Overall, this study provides valuable insights into probing and perturbation-specific strengths and weaknesses of popular multilingual Transformer-based models for different Indic languages.

Abstract

Over 46 million people worldwide suffer from
dysarthria--a motor speech disorder caused by
neurological conditions like stroke or Parkin
son's disease--making their speech slurred, un
intelligible, and emotionally distorted. This
severely affects communication, quality of life,
and social inclusion.
Wepresent the first emotion-aware framework
for dysarthric speech reconstruction, where the
speaker's emotion is detected from audio and
used to guide large language models in recover
ing intelligible, emotionally faithful sentences.
To evaluate this, we introduce a novel
metric--Multimodal
Communication
Dysarthria Score (MCDS)--which holisti
cally measures both linguistic and emotional
accuracy. Our results show strong improve
ments over traditional baselines, offering a
breakthrough toward emotionally intelligent
assistive speech systems that prioritize both
understanding and empathy

Abstract

Although speech language models are expected
to align well with brain language processing
during speech comprehension, recent studies
have found that they fail to capture brain
relevant semantics beyond low-level features.
Surprisingly, text-based language models ex
hibit stronger alignment with brain language
regions, as they better capture brain-relevant
semantics. However, no prior work has exam
ined the alignment effectiveness of text/speech
representations from multimodal models. This
raises several key questions: Can speech em
beddings from such multimodal models capture
brain-relevant semantics through cross-modal
interactions? Which modality can take advan
tage of this synergistic multimodal understand
ing to improve alignment with brain language
processing? Can text/speech representations
from such multimodal models outperform uni
modal models? To address these questions,
we systematically analyze multiple multimodal
models, extracting both text- and speech-based
representations to assess their alignment with
MEGbrain recordings during naturalistic story
listening. We find that text embeddings from
both multimodal and unimodal models signif
icantly outperform speech embeddings from
these models. Specifically, multimodal text
embeddings exhibit a peak around 200 ms, sug
gesting that they benefit from speech embed
dings, with heightened activity during this time
period. However, speech embeddings from
these multimodal models still show a similar
alignment compared to their unimodal counter
parts, suggesting that they do not gain meaning
ful semantic benefits over text-based represen
tations. These results highlight an asymmetry
in cross-modal knowledge transfer, where the
text modality benefits more from speech infor
mation, but not vice versa. We make the code
publicly available.

Abstract

We present a critical discourse analysis of the 2024 U.S. presidential debates, examining Donald Trump's rhetorical strategies in his interactions with Joe Biden and Kamala Harris. We introduce a novel annotation framework, BEADS (Bias Enriched Annotation for Dialogue Structure), which systematically extends the DAMSL framework to capture bias driven and adversarial discourse features in political communication. BEADS includes a domain and language agnostic set of tags that model ideological framing, emotional appeals, and confrontational tactics. Our methodology compares detailed human annotation with zero shot ChatGPT assisted tagging on verified transcripts from the Trump and Biden (19,219 words) and Trump and Harris (18,123 words) debates. Our analysis shows that Trump consistently dominated in key categories: Challenge and Adversarial Exchanges, Selective Emphasis, Appeal to Fear, Political Bias, and Perceived Dismissiveness. These findings underscore his use of emotionally charged and adversarial rhetoric to control the narrative and influence audience perception. In this work, we establish BEADS as a scalable and reproducible framework for critical discourse analysis across languages, domains, and political contexts.

Abstract

Music has long served as a vehicle for political expression, with protest songs playing a central role in articulating dissent and mobilizing collective action. Yet, despite their cultural significance, the linguistic and acoustic signatures that define protest music remain understudied. We present a multimodal computational analysis of protest and non-protest songs spanning multiple decades. Using NLP and audio analysis, we identify the linguistic and musical features that differentiate protest songs. Instead of focusing on classification performance, we treat classification as a diagnostic tool to investigate these features and reveal broader patterns. Protest songs are not just politically charged they are acoustically and linguistically distinct, and we quantify how.

Abstract

Generation Alpha (born 2010-2024) is the first generation fully raised within the digital ecosystem. They exhibit unique linguistic behaviours influenced by rampant online communication and platform-specific cultures. This study examines the rapid evolution of Gen Alpha slang through a comparative analysis of Millennial and Gen Z vernacular. We identify three core linguistic patterns: extreme lexical compression, digital culture-driven semantic shifts and part-of-speech conversion. We construct a comprehensive slang corpus sourced from online platforms and evaluate the performance of four AI translation systems (viz. Google Translate, ChatGPT 4, Gemini 1.0, DeepSeek v3) on over 100 slang terms. Our results reveal significant translation challenges rooted in culturally-bound terms from gaming, meme culture, and mental health discourse. Most errors are the result of inadequate cultural contextualization, with literal translations dominating the error patterns. Our findings highlight the critical limitations in current language models and emphasize the need for adaptive, culturally sensitive and context-aware frameworks that can handle the dynamic lexicon of evolving youth vernacular.

Abstract

Emotion detection in text plays a very crucial
role in NLP applications such as sentiment anal-
ysis and feedback analysis. This study tackles
two tasks: multi-label emotion detection, where
the goal is to classify text based on six emotions
(joy, sadness, fear, anger, surprise, and disgust)
in a multilingual setting, and emotion intensity
prediction, which assigns an ordinal intensity
score to each of the above-given emotions.
Using the BRIGHTER dataset, a multilingual
corpus spanning 28 languages, the paper ad-
dresses issues like class imbalances by treating
each emotion as an independent binary classifi-
cation problem. The paper first explores strate-
gies such as static embeddings such as GloVe
with logistic regression classifiers on top of it.
To capture contextual nuances more effectively,
we fine-tune transformer based models, such
as BERT and RoBERTa. Our approach demon-
strates the benefits of fine-tuning for improved
emotion prediction, while also highlighting the
challenges of multilingual and multi-label clas-
sification.