Abstract

Transformer-based language models, though not explicitly trained to mimic brain recordings, have demonstrated surprising alignment with brain activity. Progress in these models—through increased size, instruction-tuning, and multimodality—has led to better representational alignment with neural data. Recently, a new class of instruction-tuned multimodal LLMs (MLLMs) have emerged, showing remarkable zero-shot capabilities in open-ended multimodal vision tasks. However, it is unknown whether MLLMs, when prompted with natural instructions, lead to better brain alignment and effectively capture instruction-specific representations. To address this, we first investigate the brain alignment, i.e., measuring the degree of predictivity of neural visual activity using text output response embeddings from MLLMs as participants engage in watching natural scenes. Experiments with 10 different instructions (like image captioning, visual question answering, etc.) show that MLLMs exhibit significantly better brain alignment than vision-only models and perform comparably to non-instruction-tuned multimodal models like CLIP. We also find that while these MLLMs are effective at generating high-quality responses suitable to the task-specific instructions, not all instructions are relevant for brain alignment. Further, by varying instructions, we make the MLLMs encode instruction-specific visual concepts related to the input image. This analysis shows that MLLMs effectively capture count-related and recognition-related concepts, demonstrating strong alignment with brain activity. Notably, the majority of the explained variance of the brain encoding models is shared between MLLM embeddings of image captioning and other instructions. These results indicate that enhancing MLLMs' ability to capture more task-specific information could allow for better differentiation between various types of instructions, and hence improve their precision in predicting brain responses.

Abstract

Community detection is the problem of identifying natural divisions in networks. A relevant challenge in this problem is to find communities on rapidly evolving graphs. In this paper, we design efficient community detection algorithms in the batch dynamic setting. First, we present our parallel Dynamic Frontier approach. Given a batch update of edge deletions or insertions, this approach incrementally identifies an approximate set of affected vertices in the graph with minimal overhead. We apply this approach to both Louvain, a high quality, and Label Propagation Algorithm (LPA), a fast static community detection algorithm. Our approach achieves a mean speedup of 7.3× and 6.7×, when applied to Louvain and LPA respectively, compared to our parallel and optimized implementation of Δ-screening, a recently proposed state-of-the-art approach. Finally, we show how to combine Louvain and LPA with the Dynamic Frontier approach to arrive at a hybrid algorithm. This algorithm produces high-quality communities while being 14.6× faster than state-of-the-art, and identifying communities with the same quality score.

Abstract

Automated segmentation of medical image volumes promises to reduce costly medical experts’ time for annotation. However, using machine learning for the task is challenging due to variations in imaging modalities and scarcity of patient data. While interactive image segmentation methods and foundational models incorporating user-provided prompts to refine segmentation masks have shown promise, they overlook crucial sequential information between the slices in 3D medical image volumes and videos, resulting in discontinuities in the segmentation results. This paper proposes a new framework that dynamically updates model parameters during inference in a test time training framework using user-provided scribbles. Our framework preserves acquired knowledge from the previous slices of the current medical volume and the training dataset via student-teacher learning. We evaluate our method on diverse CT, MRI, and microscopic cell datasets. Our framework significantly reduces user annotation time by a factor of 6.72×. Compared to other interactive segmentation methods, we reduce the time by a factor of 2.64×. Our method also outperforms prompting foundation models for segmentation by achieving a dice score of 0.9 in 3-4 interactions compared to 5-8 user interactions for the foundation model, significantly reducing annotation time for the CT and MRI volumes.

Abstract

This paper explores the concept of solving implicature in Natural Language Processing (NLP), highlighting its significance in understanding indirect communication. Drawing on foundational theories by Austin, Searle, and Grice, we discuss how implicature extends beyond literal language to convey nuanced meanings. We review existing datasets, including the Pragmatic Understanding Benchmark (PUB), that assess models’ capabilities in recognizing and interpreting implicatures. Despite recent advances in large language models (LLMs), challenges remain in effectively processing implicature due to limitations in training data and the complexities of contextual interpretation. We propose future directions for research, including the enhancement of datasets and the integration of pragmatic reasoning tasks, to improve LLMs’ understanding of implicature and facilitate better human-computer interaction.

Abstract

This research utilises the capabilities of Google Earth Engine, a cloud-based computing platform, to conduct a comprehensive spatiotemporal assessment of the Coringa mangrove. Situated within the Coringa Wildlife Sanctuary in Andhra Pradesh, India, this mangrove – the country's second largest – is evaluated across various timeframes: 1999, 2002, 2007, 2013, 2017, and 2022. ML algorithms – Random Forest and Support Vector Machine with an RBF kernel – are employed. These algorithms analyze band composites derived from Landsat-7 ETM+ (1999, 2002, 2007), Landsat-8 (2013, 2017), and Sentinel-2 MSI (2022) satellite images. This analysis incorporates key spectral indices and utilises two spectral index thresholds for mangrove classification: one derived from established literature that identifies common thresholds at which mangroves typically occur (standard threshold), and the other is a customised threshold obtained from individual spectral index maps, tailored specifically to delineate mangrove areas in the study area (customized threshold). The results show that RF, particularly with the CT, outperforms all other methods, including RF with the ST and SVM with both thresholds, in terms of training and testing accuracy. These findings affirm the effectiveness of RF with the CT approach in accurately differentiating mangrove areas, emphasizing the critical role of threshold selection in enhancing the accuracy and competence of classification methods for mapping mangrove ecosystems.

Abstract

The significant improvements in point cloud representation learning have increased its applicability in many real-life applications, resulting in the need for lightweight, better-performing models. One widely proposed efficient method is knowledge distillation, where a lightweight model uses knowledge from large models. Very few works exist on distilling the knowledge for point clouds. Most of the work focuses on cross-modal-based approaches that make the method expensive to train. This paper proposes PointKAD, an adversarial knowledge distillation framework for point cloud-based tasks. PointKAD includes adversarial feature distillation and response distillation with the help of discriminators to extract and distill the representation of feature maps and logits. We conduct extensive experimental studies on both synthetic (ModelNet40) and real (ScanObjectNN) datasets to show that PointKAD achieves state-of-the-art results compared to the existing knowledge distillation methods for point cloud classification. Additionally, we present results on the part segmentation task, highlighting the efficacy of the PointKAD framework. Our experiments further reveal that PointKAD is capable of transferring knowledge across different tasks and datasets, showcasing its versatility. Furthermore, we demonstrate that PointKAD can be applied to a cross-modal training setup, achieving competitive performance with cross-modal-based point cloud methods for classification.

Abstract

Tornado Cash is a decentralized application (dApp) that runs on Ethereum Virtual Machine (EVM) compatible networks to enhance users’ privacy in terms of user transaction history over the blockchain. This dApp achieves its goal by enabling users to deposit currencies into designated pools and subsequently withdraw them, severing the link between depositor and withdrawer addresses. At deposit time, Tornado Cash communicates to users the level of privacy they will benefit from (anonymity set) by depositing currencies into one of its pools. Existing analyses have indicated discrepancies between the claimed anonymity set and the actual level of privacy provided, primarily attributed to users’ incorrect utilization of the dApp. This paper explores the road towards a new way to challenge the dApp’s proposed anonymity set by examining wallet fingerprints, a factor not directly related to user behavior within the application. The findings of this research shed light on the potential for creating links between clusters of users in TC according to the new proposed approach and raise a privacy concern within the Ethereum network, resulting in 13203 transactions, over 66948, linkable to the wallet used to initialize them.

Abstract

Moving Target Defense (MTD) has emerged as a proactive and dynamic framework to counteract evolving cyber threats. Traditional MTD approaches often rely on assumptions about the attacker’s knowledge and behavior. However, real-world scenarios are inherently more complex, with adaptive attackers and limited prior knowledge of their payoffs and intentions. This paper introduces a novel approach to MTD using a Markov Decision Process (MDP) model that does not rely on predefined attacker payoffs. Our framework integrates the attacker’s real-time responses into the defender’s MDP using a dynamic Bayesian Network. By employing a factored MDP model, we provide a comprehensive and realistic representation of the system and its dependencies. We also incorporate incremental updates to an attack response predictor as new data emerges, ensuring an adaptive and robust defense mechanism. Additionally, we consider the costs of switching configurations in MTD, integrating them into the reward structure to balance execution and defense costs. We first highlight the challenges of the problem through a theoretical negative result on regret. However, our empirical evaluations, conducted in network and web application settings, demonstrate the framework’s effectiveness in environments characterized by high uncertainty and dynamically changing attack surfaces.

Abstract

Text driven diffusion models have shown remarkable capabilities in editing images. However, when editing 3D scenes, existing works mostly rely on training a NeRF for 3D editing. Recent NeRF editing methods leverages edit operations by deploying 2D diffusion models and project these edits into 3D space. They require strong positional priors alongside text prompt to identify the edit location. These methods are operational on small 3D scenes and are more generalized to particular scene. They require training for each specific edit and cannot be exploited in real-time edits. To address these limitations, we propose a novel method, FreeEdit, to make edits in training free manner using mesh representations as a substitute for NeRF. Training-free methods are now a possibility because of the advances in foundation model’s space. We leverage these models to bring a training-free alternative and introduce solutions for insertion, replacement and deletion. We consider insertion, replacement and deletion as basic blocks for performing intricate edits with certain combinations of these operations. Given a text prompt and a 3D scene, our model is capable of identifying what object should be inserted/replaced or deleted and location where edit should be performed. We also introduce a novel algorithm as part of FreeEdit to find the optimal location on grounding object for placement. We evaluate our model by comparing it with baseline models on a wide range of scenes using quantitative and qualitative metrics and showcase the merits of our method with respect to others.

Abstract

No Results Found