Abstract

Wikipedia is a highly essential platform because of its informative, dynamic, and easily accessible nature. To identify topics/titles warranting their own Wikipedia article, editors of Wikipedia defined “Notability” guidelines. So far notability is enforced by humans, which makes scalability an issue. There has been no significant work on Notability determination for titles with complex category dependencies. We design a mechanism to identify such titles. We construct a dataset with 9k such titles and propose a category-agnostic approach utilizing Graph neural networks, for their notability determination. Our system outperforms machine learning-based, transformer-based classifiers and entity salience methods. It provides a scalable alternative to manual decision-making about notability.

Abstract

Knowledge graphs (KGs) have been playing a crucial role in leveraging informa- tion on web for several downstream tasks, making it vital to construct and maintain them. Despite previous efforts in populating KGs, these methods typically do not focus on analyzing entity-specific content exclusively but rely on a fixed collection of documents. We define an approach to populate such KGs by utilizing entity-specific content on the web, for generating entity embeddings to establish entity-category interconnections. We empirically prove our ap- proach’s effectiveness, by utilizing it for a downstream task of Notability detection, associated with one of the most popular and important Knowledge Graphs - Wikipedia platform. To mod- erate the content uploaded to Wikipedia, “Notability” guidelines are defined by its editors to identify named entities that warrant their article on Wikipedia. So far notability is enforced by humans, which makes scalability an issue, and there has been no significant work on automat- ing this process. In this paper, we define a multipronged category-agnostic approach based on web-based entity features and their text-based salience encodings, to construct entity embed- dings for determining an entity’s notability. We distinguish entities based on their categories and utilize neural networks to perform classification. For validation, we utilize accuracy and prediction confidence on popular Wikipedia pages. Our system outperforms machine learning- based classifier approaches and handcrafted entity salience detection algorithms, by achieving performance accuracy of around 88%. Our system provides an efficient and scalable alterna- tive to manual decision-making about the importance of a topic, which could be extended to other such KG-based tasks

Abstract

This paper describes our approach for SemEval- 2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense. The BRAIN- TEASER task comprises multiple-choice Ques- tion Answering designed to evaluate the mod- els’ lateral thinking capabilities. It consists of Sentence Puzzle and Word Puzzle subtasks that require models to defy default common- sense associations and exhibit unconventional thinking. We propose a unique strategy to im- prove the performance of pre-trained language models, notably the Gemini 1.0 Pro Model, in both subtasks. We employ static and dy- namic few-shot prompting techniques and in- troduce a model-generated reasoning strategy that utilizes the LLM’s reasoning capabilities to improve performance. Our approach demon- strated significant improvements, showing that it performed better than the baseline models by a considerable margin but fell short of per- forming as well as the human annotators, thus highlighting the efficacy of the proposed strate- gies. We have made our code open-sourced for the replicability of our methods.

Abstract

Architectural Knowledge Management (AKM) involves the organized handling of information related to architectural decisions and design within a project or organization. An essential artefact of AKM is the Architecture Decision Records (ADR), which documents key design decisions. ADRs are documents that capture decision context, decision made and various aspects related to a design decision, thereby promoting transparency, collaboration, and understanding. Despite their benefits, ADR adoption in software development has been slow due to challenges like time constraints and inconsistent uptake. Recent advancements in Large Language Models (LLMs) may help bridge this adoption gap by facilitating ADR generation. However, the effectiveness of LLM for ADR generation or understanding is something that has not been explored. To this end, in this work, we perform an exploratory study which aims to investigate the feasibility of using LLM for the generation of ADRs given the decision context. In our exploratory study, we utilize GPT and T5-based models with 0-shot, few-shot, and fine-tuning approaches to generate the Decision of an ADR given its Context. Our results indicate that in a 0-shot setting, state-of-the-art models such as GPT-4 generate relevant and accurate Design Decisions, although they fall short of human-level performance. Additionally, we observe that more cost-effective models like GPT-3.5 can achieve similar outcomes in a few-shot setting, and smaller models such as Flan-T5 can yield comparable results after fine-tuning. To conclude, this exploratory study suggests that LLM can generate Design Decisions, but further research is required to attain human-level generation and establish standardized widespread adoption.

Abstract

Commonsense question-answering (QA) methods combine the power of pre-trained Language Models (LM) with the reasoning provided by Knowledge Graphs (KG). A typical approach collects nodes relevant to the QA pair from a KG to form a Working Graph (WG) followed by reasoning using Graph Neural Networks (GNNs). This faces two major challenges: (i) it is difficult to capture all the information from the QA in the WG, and (ii) the WG contains some irrelevant nodes from the KG. To address these, we propose GrapeQA with two simple improvements on the WG: (i) Prominent Entities for Graph Augmentation identifies relevant text chunks from the QA pair and augments the WG with corresponding latent representations from the LM, and (ii) ContextAware Node Pruning removes nodes that are less relevant to the QA pair. We evaluate our results on OpenBookQA, CommonsenseQA and MedQA-USMLE and see that GrapeQA shows consistent improvements over its LM + KG predecessor (QA-GNN in particular) and large improvements on OpenBookQA.

Abstract

Augmenting Wikidata with spatial relations specific to Geography can be useful for increasing its utility in multiple applications. In this paper, we aim to extract orientation of borders between countries in the world, from their Wikipedia text and suggest its use to augment the shares_borders_with relation in Wikidata. We propose the use of Natural Language Inference (NLI) for extracting the orientation relations from text and show that when combined with contextual lexical patterns, the performance becomes better than the standard NLI setting.

Abstract

Fact-to-text generation can allow for the generation of high-quality, informative texts such as Wikipedia articles. Cross-lingual fact-to-text generation (XF2T) involves using facts available in a language, typically English, and generating texts in a different language based on these facts. This is particularly relevant for low and medium-resource languages, which have relatively structured informative content. This work explores the problem of XF2T for generating long text from given facts with a specific focus on generating factually grounded content. Unfortunately, previous work either focuses on cross-lingual facts to short text or monolingual graph to text generation. In this paper, we propose a novel solution to the multi-sentence XF2T task, which addresses these challenges by training multilingual Transformer-based models with coverage prompts and rebalanced beam search, and further improving the quality by defining task-specific reward functions and training on them using reinforcement learning. Keywords: XF2T, text generation, cross-lingual, NLG evaluation, low resource NLG

Abstract

This paper describes our system (iREL) for Tweet intimacy analysis shared task of the SemEval 2023 workshop at ACL 2023. Our system achieved an overall Pearson’s r score of 0.5924 and ranked 10th on the overall leaderboard. For the unseen languages, we ranked third on the leaderboard and achieved a Pearson’s r score of 0.485. We used a single multilingual model for all languages, as discussed in this paper. We provide a detailed description of our pipeline along with multiple ablation experiments to further analyse each component of the pipeline. We demonstrate how translation-based augmentation, domainspecific features, and domain-adapted pretrained models improve the understanding of intimacy in tweets. The code can be found at https://github.com/bhavyajeet/Multilingualtweet-intimacy

Abstract

Factual consistency is one of the most important requirements when editing high quality documents. It is extremely important for automatic text generation systems like summarization, question answering, dialog modeling, and language modeling. Still, automated factual inconsistency detection is rather under-studied. Existing work has focused on (a) finding fake news keeping a knowledge base in context, or (b) detecting broad contradiction (as part of natural language inference literature). However, there has been no work on detecting and explaining types of factual inconsistencies in text, without any knowledge base in context. In this paper, we leverage existing work in linguistics to formally define five types of factual inconsistencies. Based on this categorization, we contribute a novel dataset, FICLE (Factual Inconsistency CLassification with Explanation), with ~8K samples where each sample consists of two sentences (claim and context) annotated with type and span of inconsistency. When the inconsistency relates to an entity type, it is labeled as well at two levels (coarse and fine-grained). Further, we leverage this dataset to train a pipeline of four neural models to predict inconsistency type with explanations, given a (claim, context) sentence pair. Explanations include inconsistent claim fact triple, inconsistent context span, inconsistent claim component, coarse and fine-grained inconsistent entity types. The proposed system first predicts inconsistent spans from claim and context; and then uses them to predict inconsistency types and inconsistent entity types (when inconsistency is due to entities). We experiment with multiple Transformer …

Abstract

Named Entity Recognition(NER) is a task of recognizing entities at a token level in a sen- tence. This paper focuses on solving NER tasks in a multilingual setting for complex named entities.Our team, LLM-RM partici- pated in the recently organized SemEval 2023 task, Task 2: MultiCoNER II,Multilingual Complex Named Entity Recognition. We approach the problem by leveraging cross- lingual representation provided by fine-tuning XLM-Roberta base model on datasets of all of the 12 languages provided - Bangla, Chinese, English, Farsi, French, German, Hindi, Italian, Portuguese, Spanish, Swedish and Ukrainian