Abstract
Coreference resolution (CR) is the task of identifying text spans that refer to the same entity. It is a
fundamental component of natural language understanding with applications in various downstream
NLP tasks, such as question answering, knowledge graph construction, and summarization. Despite
its significance and the advancements made by neural coreference models, CR models face a major
bottleneck: their limited generalization capability.
Prior work attributes this generalization gap to differences in annotations, such as what constitutes a
mention (or entity) and varying preferences to span boundaries. For a model to have strong referential
capabilities, it must adapt to these annotation-specific nuances. However, achieving this level of
adaptability remains a significant challenge, even for state-of-the-art (SOTA) models. This challenge
is further amplified when evaluating the referential capabilities of large language models (LLMs) in a
few-shot setting, where replicating nuanced annotations with just a few examples is highly unrealistic.
We observe that these annotation-specific nuances, can be beneficial but are not essential for downstream
tasks or for evaluating the core referential capabilities of an LLM. We describe these nuances as bells
and whistles.
In this work, we redefine the traditional formulation of coreference resolution by shifting focus
away from its bells and whistles. Instead, we propose task formulations more aligned with practical
applications and demonstrate improved generalizability across domains.
Our first contribution introduces an alternative referential task, Major Entity Identification (MEI). MEI
simplifies referential tasks by:(a) assuming that target entities are explicitly provided in the input, and (b)
focusing exclusively on frequent entities. Assuming entities to be part of the input shifts the responsibility
for domain-specific annotation adaptation—determining which entities are annotated—from the training
phase to inference. Through extensive experiments, we show that MEI models generalize effectively
across domains using both supervised approaches and LLM-based few-shot prompting across multiple
datasets. Importantly, MEI aligns with the classification framework, enabling the use of robust, intuitive,
and well-understood classification-based evaluation metrics. Beyond its theoretical appeal, MEI also has
practical utility as it allows users to efficiently search for all mentions of a specific entity or a group of
entities of interest. Our second major contribution addresses critical shortcomings identified in recent evaluations of
large language models (LLMs) on coreference resolution. These studies revealed that traditional output
formats and evaluation metrics fail to capture models’ referential understanding fully. Traditional
evaluation methods require reproducing the entire document along with annotated cluster information or
precisely replicating the antecedent span. This introduces additional bells and whistles, such as ensuring
the accurate reproduction of spans and documents. To tackle this issue, we introduce IdentifyMe,
a new benchmark for mention resolution that adopts a multiple-choice question (MCQ) format—a
widely used evaluation approach for LLMs. With this simplified task design, any failure can now, be
attributed exclusively to issues with mention resolution. IdentifyMe presents long narratives and applies
heuristics to eliminate easily identifiable mentions, resulting in a more challenging and rigorous task. The
benchmark incorporates a curated mix of various mention types and their corresponding entities, enabling
fine-grained analysis of model performance. Notably, LLM performance remains substantially below
human-level performance on IdentifyMe, highlighting considerable room for improvement even for
advanced models like GPT-4. The evaluation also reveals key weaknesses in current LLMs, particularly
with pronominal mentions, nested mentions, and other nuanced cases.
Overall, this work moves beyond traditional coreference resolution formulations, focusing on tasks
with practical applicability and providing fresh insights into the referential strengths and weaknesses
of current models. We term this approach Coreference Without Bells and Whistles — a streamlined
perspective that prioritizes utility and understanding of model capabilities over tailored annotation
adaptation