Abstract

This study presents efforts focused on extracting and structuring doctor notes, specifically Magnetic Resonance Imaging (MRI) reports, into a standardized format using large language models (LLMs). We introduce a novel benchmark dataset comprising of 55 clinically relevant variables given by doctors, making it the first of its kind in the automated processing of unstructured medical texts. The annotations to the dataset were generated using a systematic prompt-tuning approach that was manually validated. It was then evaluated across three experimental stages: baseline, intermediate, and fine-tuned. Each stage assessed the impact of different prompt strategies on the performance of various LLMs (LLaMA, Qwen, and DeepSeek). Among the models tested, LLaMA 3.1 8B Instruct consistently achieved the highest composite score in both the intermediate and final phases, resulting in an 18.42% improvement in performance.

Abstract

Extracting structured clinical data from unstructured patient transcripts is challenging due to large target
schemas and inherent linguistic ambiguity. We address the extraction of 193 heterogeneous clinical
attributes from nursing notes and clinician-patient dialogues, and demonstrate that zero-shot large
language models (LLMs) are ineffective in this setting, achieving an F1 score below 0.15 due to context
window saturation and hallucination. We propose a four-stage framework that combines semantic
schema clustering, role-based chain-of-thought prompting, supervised fine-tuning of Llama-3.1-8B, and
transcript-verified post-processing. Our approach achieves an F1 score of 0.66, representing a 4.4x
improvement over the baseline, by balancing high recall from generative models with high precision
from verification. These results highlight the effectiveness of hybrid pipelines for high-stakes clinical
information extraction.

Abstract

Recent advances in LLM agentic systems have improved the
automation of offensive security tasks, particularly for Capture
the Flag (CTF) challenges. We systematically investigate the
key factors that drive agent success and provide a detailed
recipe for building effective LLM-based offensive security
agents. First, we present CTFJudge, a framework leverag-
ing LLM as a judge to analyze agent trajectories and provide
granular evaluation across CTF solving steps. Second, we pro-
pose a novel metric, CTF Competency Index (CCI) for partial
correctness, revealing how closely agent solutions align with
human-crafted gold standards. Third, we examine how LLM
hyperparameters, namely temperature, top-p, and maximum
token length, influence agent performance and automated cy-
bersecurity task planning. For rapid evaluation, we present
CTFTiny, a curated benchmark of 50 representative CTF
challenges across binary exploitation, web, reverse engineer-
ing, forensics, and cryptography. Our findings identify optimal
multi-agent coordination settings and lay the groundwork for
future LLM agent research in cybersecurity.

Abstract

Large language models are rapidly changing how
learners acquire and demonstrate cybersecurity skills. However,
when human-AI collaboration is allowed, educators still lack val-
idated competition designs and evaluation practices that remain
fair and evidence-based. This paper presents a cross-regional
study of LLM-centered Capture-the-Flag competitions built on
the Cyber Security Awareness Week competition system. To un-
derstand how autonomy levels and participants' knowledge back-
grounds influence problem-solving performance and learning-
related behaviors, we formalize three autonomy levels: human-in-
the-loop, autonomous agent frameworks, and hybrid. To enable
verification, we require traceable submissions including conver-
sation logs, agent trajectories, and agent code. We analyze multi-
region competition data covering an in-class track, a standard
track, and a year-long expert track, each targeting participants
with different knowledge backgrounds. Using data from the 2025
competition, we compare solve performance across autonomy
levels and challenge categories, and observe that autonomous
agent frameworks and hybrid achieve higher completion rates
on challenges requiring iterative testing and tool interactions.
In the in-class track, we classify participants' agent designs and
find a preference for lightweight, tool-augmented prompting and
reflection-based retries over complex multi-agent architectures.
Our results offer actionable guidance for designing LLM-assisted
cybersecurity competitions as learning technologies, including
autonomy-specific scoring criteria, evidence requirements that
support solution verification, and track structures that improve
accessibility while preserving reliable evaluation and engagement.

Abstract

We present a system for the MEDIQA-SYNUR 2026 shared task on extracting structured clinical observations from nurse dictation transcripts. The transcripts contain spoken-style clinical language with disfluencies, filler words, and hesitations. Our approach is a four-stage LLM inference pipeline preceded by an offline knowledge enhancement step: (1) knowledge-enhanced concept detection using medical domain clustering, (2) evidence-grounded value extraction, (3) schema-constrained value normalization, and (4) deterministic post-processing with fuzzy matching and unit pairing. In the offline step, we use the task ontology and training examples to generate per-concept clinical definitions and extraction rules, and group the 193 concepts into 19 non-exclusive medical domain clusters. These are injected into all downstream prompts as domain priors. All LLM stages use gpt-oss-120b with structured JSON output and chain-of-thought reasoning. The task requires exact matching on concept ID and value pairs across a 193-concept ontology, making precision particularly challenging. We iteratively refine concept definitions and prompt guidelines based on error analysis of the training data. Our system achieves an F1 score of 0.806 on the test set.

Abstract

Despite negation being one of the core element in any language, it remains a challenging phenomenon for modern Large Language Models(LLMs). Recently, there have been growing efforts to evaluate how models handle negation. However, the existing probing datasets are mostly English-centric. To facilitate evaluation for Indian Languages especially Telugu, which has complex morphological features, we present NEGTEG benchmark. This benchmark is a test suite that contains 5 tasks: Negation Detection, Negation Translation, Paraphrase Detection, Sentiment Analysis and Polarity Flipping. The test suite is designed based on strong linguistic analysis and includes annotations of different negation types. This helps us evaluate how models perform across various forms of negation. We use the benchmark to probe the negation handling capabilities of multilingual language models at different levels and our evaluation reveals that most of the models struggle significantly with Telugu negation across all tasks.

Abstract

Discourse-level translation remains a challenge for machine translation (MT) systems, particularly for translation from Indian languages to English. This challenge is amplified in Indian languages due to rich morphology and discourse complexity. Existing evaluation benchmarks prioritize sentence-level translation quality and fail to capture important discourse phenomena such as pronoun resolution and lexical cohesion. To address this gap, we introduce IndicDISCO-MT, a parallel benchmark dataset covering translations from eight Indian languages, namely: Bengali, Gujarati, Hindi, Marathi, Kannada, Tamil, Telugu and Urdu to English. On top of this corpus, we introduce DiscoAlign, a human-annotated source-to-target alignment pairs for discourse evaluation. We further propose two evaluation benchmarks, ProAlign and LexiAlign, to evaluate ability of large language models (LLMs) and MT systems to handle personal pronouns and lexical cohesion. Our evaluation of recent LLM and MT systems shows that although models achieve high translation quality, they still struggle to accurately preserve discourse-level phenomena. The proposed benchmarks provide a systematic framework for evaluating discourse-aware translation and can facilitate the development of more coherent and contextually consistent translations. The dataset is made publicly available.

Abstract

Product reviews on e-commerce platforms are a critical form of user-generated content that influence consumer decisions. However, these reviews are predominantly in English, creating a significant accessibility barrier for users who are not fluent in English. When translating into major Indian languages using the current models, the outputs often fail to capture domain-specific features and colloquial style, resulting in stylistically unnatural texts. To address this gap, we introduce textbf{STAR-IL}, a human-annotated, multilingual, parallel corpus for style-aware translation of product reviews. We evaluate the performance of several state-of-the-art models on our dataset for the task of product review translation. Our experiments show that models fine-tuned on STAR-IL achieve significant average performance gain of textbf{5.77} points in BLEU and textbf{3.78} points in COMET, when compared to their baselines, across all languages. Our dataset provides a valuable benchmark for future research in style-aware product review translation. The STAR-IL dataset is publicly available at https://github.com/ltrc/STAR-IL-Corpus

Abstract

The extent to which large language models (LLMs) can perform culturally grounded reasoning across non-English languages remains underexplored. This paper examines the reasoning and self-assessment abilities of LLMs across seven major Indian languages- Bengali, Gujarati, Hindi, Kannada, Malayalam, Tamil, and Telugu. We introduce a multilingual riddle dataset combining traditional riddles with context-reconstructed variants and evaluate five LLMs- Gemini 2.5 Pro, Gemini 2.5 Flash, Mistral-Saba, LLaMA-4-Scout, and LLaMA-4-Maverick under seven prompting strategies. In the first stage, we assess riddle-solving performance and find that while Gemini 2.5 Pro performs best overall, few-shot methods yield only marginal gains, and accuracy varies notably across languages. In the second stage, we conduct a self-evaluation experiment to measure reasoning consistency. The results reveal a key finding: a model's initial accuracy is inversely correlated with its ability to identify its own mistakes. Top-performing models such as Gemini 2.5 Pro are overconfident (4.34% True Negative Rate), whereas lower-performing models like LLaMA-4-Scout are substantially more self-aware (42.09% True Negative Rate). These results point to clear gaps in multilingual reasoning and highlight the need for models that not only reason effectively but also recognize their own limitations.

Abstract

Healthcare community question-answering (CQA) forums have become popular for users seeking medical advice, offering answers that range from personal experiences to factual information. Traditionally, CQA summarization relies on the best-voted answer as a reference summary. However, this approach overlooks the diverse perspectives across multiple responses. Structuring summaries by perspective could better meet users' informational needs. The PerAnsSumm shared task addresses this by identifying and classifying perspective-specific spans (Task_A) and generating perspective-specific summaries from question-answer threads (Task_B). In this paper, we present our work on the PerAnsSumm shared task 2025 at the CL4Health Workshop, NAACL 2025. Our system leverages the RoBERTa-large model for identifying perspective-specific spans and the BART-large model for summarization. We achieved a Macro-F1 score of 0.9 (90%) and a Weighted-F1 score of 0.92 (92%) for classification. For span matching, our strict matching F1 score was 0.21 (21%), while proportional matching reached 0.68 (68%), resulting in an average Task A score of 0.6 (60%). For Task B, we achieved a ROUGE-1 score of 0.4 (40%), ROUGE-2 of 0.18 (18%), and ROUGE-L of 0.36 (36%). Additionally, we obtained a BERTScore of 0.84 (84%), METEOR of 0.37 (37%), and BLEU of 0.13 (13%), resulting in an average Task B score of 0.38 (38%). Combining both tasks, our system achieved an overall average score of 49% and ranked 6th on the official leaderboard for the shared task.