Abstract

A Message Sequence Chart (MSC) is a visually comprehensible knowledge representation used for showing events of a narrative in their correct temporal order. However, to enable its use as a spatio-temporal knowledge representation, it needs to be enhanced with spatial knowledge present in the narrative. In this work, we aim at enriching MSCs with spatial knowledge, using their condition construct, to enable them for downstream spatio-temporal applications. We propose two Large Language Model (LLM) based approaches for MSC construction, one based on few-shot prompting and another involving fine-tuning. Further, we develop a few-shot prompting approach to enable the spatial enrichment of the constructed MSCs. We report on performance of the proposed approaches and also demonstrate the utility of the enrichment in an image-based visualization application.

Abstract

A Message Sequence Chart (MSC) is a visually comprehensible knowledge representation used for showing events of a narrative in their correct temporal order. However, to enable its use as a spatio-temporal knowledge representation, it needs to be enhanced with spatial knowledge present in the narrative. In this work, we aim at enriching MSCs with spatial knowledge about locations of the involved entities, using their condition construct, to enable them for downstream spatio-temporal applications. We propose two Large Language Model (LLM) based approaches for MSC construction, one based on few-shot in-context learning and another involving fine-tuning. Further, we propose a new task of motion reasoning, where an entity's location is reasoned based on the last event it was part of and whether the event involved any motion causing change of the entity's location. Similar as earlier, we develop a few-shot in-context learning approach and an LLM fine-tuning in-context learning to solve the motion reasoning task and enable the spatial enrichment of the constructed MSCs. We report performance of the proposed approaches and also demonstrate the utility of the enrichment in two applications on image-based visualization and narrative gap finding.

Abstract

Hallucinations in large language models(LLMs) have recently become a significantproblem. A recent effort in this directionis a shared task at Semeval 2024 Task 6,SHROOM, a Shared-task on Hallucinationsand Related Observable Overgeneration Mis-takes. This paper describes our winning so-lution ranked 1st and 2nd in the 2 sub-tasksof model agnostic and model aware tracks re-spectively. We propose a meta-regressor basedensemble of LLMs based on a random forestalgorithm that achieves the highest scores onthe leader board. We also experiment with var-ious transformer based models and black boxmethods like ChatGPT, Vectara, and others. Inaddition, we perform an error analysis com-paring ChatGPT against our best model whichshows the limitations of the former

Abstract

Wikipedia is a highly essential platform because of its informative, dynamic, and easily accessible nature. To identify topics/titles warranting their own Wikipedia article, editors of Wikipedia defined "Notability" guidelines. So far notability is enforced by humans, which makes scalability an issue. There has been no significant work on Notability determination for titles with complex category dependencies. We design a mechanism to identify such titles. We construct a dataset with 9k such titles and propose a category-agnostic approach utilizing Graph neural networks, for their notability determination. Our system outperforms machine learning-based, transformer-based classifiers and entity salience methods. It provides a scalable alternative for notability detection.

Abstract

Curriculum learning has been used to improve the quality of text generation systems by ordering the training samples according to a particular schedule in various tasks. In the context of data-to-text generation (DTG), previous studies used various difficulty criteria to order the training samples for monolingual DTG. These criteria, however, do not generalize to the crosslingual variant of the problem and do not account for noisy data. We explore multiple criteria that can be used for improving the performance of cross-lingual DTG systems with noisy data using two curriculum schedules. Using the alignment score criterion for ordering samples and an annealing schedule to train the model, we show increase in BLEU score by up to 4 points, and improvements in faithfulness and coverage of generations by 5-15% on average across 11 Indian languages and English in 2 separate datasets. We make code and data publicly available

Abstract

A visual question answering (VQA) system for electrical circuit images could be useful as a quiz generator, design and verification assistant or an electrical diagnosis tool. Although there exists a vast literature on VQA, to the best of our knowledge, there is no existing work on VQA for electrical circuit images. To this end, we curate a new dataset, CircuitVQA, of 115K+ questions on 5725 electrical images with
70 circuit symbols. The dataset contains schematic as well as hand-drawn images. The questions span various categories like counting, value, junction and position based questions. To be effective, models must demonstrate skills like object detection, text recognition, spatial understanding, question intent understanding and answer generation. We experiment with multiple foundational visio-linguistic models for this task and find that a finetuned BLIP model with component descriptions as additional input provides best results. We make the code and dataset publicly available

Abstract

To employ Pre-trained Language Models (PLMs) as knowledge containers in niche domains it is important to gauge the knowledge of these PLMs about facts in these domains. It is also an important pre-requisite to know how much enrichment effort is required to make them better. As part of this work, we aim to gauge and enrich small PLMs for knowledge of world geography. Firstly, we develop a moderately sized dataset of masked sentences covering 24 different fact types about world geography to estimate knowledge of PLMs on these facts. We hypothesize that for this niche domain, smaller PLMs may not be well equipped. Secondly, we enrich PLMs with this knowledge through fine-tuning and check if the knowledge in the dataset is infused sufficiently. We further hypothesize that linguistic variability in the manual templates used to embed the knowledge in masked sentences does not affect the knowledge infusion. Finally, we demonstrate the application of PLMs to tourism blog search and Wikidata KB augmentation. In both applications, we aim at showing the effectiveness of using PLMs to achieve competitive performance.

Abstract

Traditional evaluation metrics like BLEU and ROUGE fall short when capturing the nuanced qualities of generated text, particularly when there is no single ground truth. In this paper, we explore the potential of Large Language Models (LLMs), specifically Google Gemini 1, to serve as automatic evaluators for non-standardized metrics in summarization and dialog-based tasks. We conduct experiments across multiple prompting strategies to examine how LLMs fare as quality evaluators when compared with human judgments on the SummEval and USR datasets, asking the model to generate both a score as well as a justification for the score. Furthermore, we explore the robustness of the LLM evaluator by using perturbed inputs. Our findings suggest that while LLMs show promise, their alignment with human evaluators is limited, they are not robust against perturbations and significant improvements are required for their standalone use as reliable evaluators for subjective metrics.

Abstract

While documenting Architectural Knowledge (AK) is crucial, it is frequently neglected in many projects, and existing manual tools are underutilized. Although undocumented, Archi-tecture Knowledge (AK) is dispersed across various sources such as source code, documentation, and runtime logs. To address this, automated tools for efficient AK extraction and documentation are essential. Even after generating AK, navigating through vast the Architectural Records can be overwhelming. Building on that, we propose an automated Architectural Knowledge Management (AKM) System using Information Extraction and Generative AI, which generates AK from various source for a given system and answers architectural queries with respect to the given system. The development of an efficient Architectural Knowledge Management (AKM) system, which is both effective and user-friendly, entails the resolution of numerous challenges. It requires consolidating diverse AK data sources scattered across code, dia-grams, repository commits, and online platforms. The integration of Multimodal AI for AK extraction, incorporation of global AK, and leveraging Generative AI for AK documentation further compounds the problem. Moreover, generating contextually appropriate query responses adds another layer of complexity. To this end, we performed an initial exploratory study on generating Architectural Design Decisions using generative Large Language Models (LLM) in the context of Architecture Decision Records (ADR). Our initial results have been promising indicating the potential impact of GenAI for architectural knowledge management.

Abstract

Architectural Knowledge Management (AKM) involves the organized handling of information related to architectural decisions and design within a project or organization. An essential artefact of AKM is the Architecture Decision Records (ADR), which documents key design decisions. ADRs are documents that capture decision context, decision made and various aspects related to a design decision, thereby promoting transparency, collaboration, and understanding. Despite their benefits, ADR adoption in software development has been slow due to challenges like time constraints and inconsistent uptake. Recent advancements in Large Language Models (LLMs) may help bridge this adoption gap by facilitating ADR generation. However, the effectiveness of LLM for ADR generation or understanding is something that has not been explored. To this end, in this work, we perform an exploratory study which aims to investigate the feasibility of using LLM for the generation of ADRs given the decision context. In our exploratory study, we utilize GPT and T5-based models with 0-shot, few-shot, and fine-tuning approaches to generate the Decision of an ADR given its Context. Our results indicate that in a 0-shot setting, state-of-the-art models such as GPT-4 generate relevant and accurate Design Decisions, although they fall short of human-level performance. Additionally, we observe that more cost-effective models like GPT-3.5 can achieve similar outcomes in a few-shot setting, and smaller models such as Flan-T5 can yield comparable results after fine-tuning. To conclude, this exploratory study suggests that LLM can generate Design Decisions, but further research is required to attain human-level generation and establish standardized widespread adoption.