Abstract

Current Non-Audible Murmur (NAM)-to-speech techniques rely on voice cloning to simulate ground-truth speech from paired whispers. However, the simulated speech often lacks intelligibility and fails to generalize well across different speakers. To address this issue, we focus on learning phoneme-level alignments from paired whispers and text and employ a Textto-Speech (TTS) system to simulate the ground-truth. To reduce dependence on whispers, we learn phoneme alignments directly from NAMs, though the quality is constrained by the available training data. To further mitigate reliance on NAM/whisper data for ground-truth simulation, we propose incorporating the lip modality to infer speech and introduce a novel diffusionbased method that leverages recent advancements in lip-tospeech technology. Additionally, we release the MultiNAM dataset with over 7.96 hours of paired NAM, whisper, video, and text data from two speakers and benchmark all methods on this dataset. Speech samples and the dataset are available at https://diff-nam.github.io/DiffNAM/

Abstract

Image captioning systems are unable to generate fine-grained captions as they are trained on data that is either noisy (alt-text) or generic (human annotations). This is further exacerbated by maximum likelihood training that encourages generation of frequently occurring phrases. Previous works have tried to address this limitation by fine-tuning captioners with a self-retrieval (SR) reward. However, we find that SR fine-tuning has a tendency to reduce caption faithfulness and even hallucinate. In this work, we circumvent this bottleneck by improving the MLE initialization of the captioning system and designing a curriculum for the SR fine-tuning process. To this extent, we present (1) Visual Caption Boosting, a novel framework to instill fine-grainedness in generic image captioning datasets while remaining anchored in human annotations; and (2) BagCurri, a carefully designed training curriculum that more optimally leverages the contrastive nature of the self-retrieval reward. Jointly, they enable the captioner to describe fine-grained aspects in the image while preserving faithfulness to ground-truth captions. Our approach outperforms previous work by +8.9% on SR against 99 random distractors (RD100) (Dessi et al., 2023); and +7.6% on ImageCoDe. Additionally, existing metrics to evaluate captioning systems fail to reward diversity or evaluate a model's fine-grained understanding ability. Our third contribution addresses this by proposing self-retrieval from the lens of evaluation. We introduce TrueMatch, a benchmark comprising bags of highly similar images that uses SR to assess the captioner's ability to capture subtle visual distinctions. We evaluate and compare several state-of-the-art open-source MLLMs on TrueMatch, and find that our SR approach outperforms them all by a significant margin (e.g. +4.8% - 7.1% over Cambrian) while having 1-2 orders of magnitude fewer parameters. We also outperform vanilla SR by +14.4% to +19.5%.

Abstract

In the Himalayan region, traditional buildings made of masonry have existed for centuries and still make up a sizable portion of the building stock. Many of these buildings have proven records of earthquake resilience. At the same time, the primary cause of fatalities from previous earthquakes is the increasing collapse of unreinforced masonry structures. In various Himalayan locations, local seismic culture has evolved over the past few centuries to improve the seismic capacity of stone constructions. These areas use wood or other materials with high tensile strength to reinforce their stone masonry. The post-earthquake reconnaissance surveys clearly demonstrate the superior performance of these timber-reinforced masonry constructions. Nevertheless, there is still a lack of scientific investigation, either through numerical modelling or laboratory-based studies, on the behaviour and mechanism of these structures. Furthermore, there are no explicit criteria for evaluating timber-reinforced masonry in the present Indian seismic code. This study examines several distinct traditional masonry building typologies found in the Himalayas, such as ‘Thathara’, ‘Kath-Kuni’ or ‘Koti-Banal’, ‘Newari’, ‘Dhajji Dewari’, ‘Ikra’ or ‘Assam-Type House’, ‘Bhatar’ or ‘Taq’, and ‘Dry-stone buildings’. For the majority of these typologies, aside from Dry-stone houses, the masonry walls have embedded horizontal timber bands or lacings that enhance box behaviour, thereby preventing premature failure. Through a thorough analysis of timber-reinforced masonry, including architectural details, earthquake-resilient & vulnerable features, and experimental and numerical investigations, the article presents a unique addition to our understanding of seismic performance and how it might best protect the Himalayan region’s traditional heritage. Additionally, the current study contributes to a better understanding of how wood elements give masonry structures greater seismic resilience. This understanding will help with several initiatives: (i) better conservation, preservation, and maintenance of traditional or heritage structures; (ii) seismic codes/standards providing details regarding the assessment of timber-reinforced structures; (iii) supporting and encouraging practitioners to design and build using techniques that have demonstrated superior performance, and (iv) as attempts to lessen the impact of building on our environment increases, provide a way to make safer structures utilizing low-carbon materials.

Abstract

Community detection is the problem of identifying natural divisions in networks. A relevant challenge in this problem is to find communities on rapidly evolving graphs. In this paper, we design efficient community detection algorithms in the batch dynamic setting. First, we present our parallel Dynamic Frontier approach. Given a batch update of edge deletions or insertions, this approach incrementally identifies an approximate set of affected vertices in the graph with minimal overhead. We apply this approach to both Louvain, a high quality, and Label Propagation Algorithm (LPA), a fast static community detection algorithm. Our approach achieves a mean speedup of 7.3× and 6.7×, when applied to Louvain and LPA respectively, compared to our parallel and optimized implementation of Δ-screening, a recently proposed state-of-the-art approach. Finally, we show how to combine Louvain and LPA with the Dynamic Frontier approach to arrive at a hybrid algorithm. This algorithm produces high-quality communities while being 14.6× faster than state-of-the-art, and identifying communities with the same quality score.

Abstract

Automated segmentation of medical image volumes promises to reduce costly medical experts’ time for annotation. However, using machine learning for the task is challenging due to variations in imaging modalities and scarcity of patient data. While interactive image segmentation methods and foundational models incorporating user-provided prompts to refine segmentation masks have shown promise, they overlook crucial sequential information between the slices in 3D medical image volumes and videos, resulting in discontinuities in the segmentation results. This paper proposes a new framework that dynamically updates model parameters during inference in a test time training framework using user-provided scribbles. Our framework preserves acquired knowledge from the previous slices of the current medical volume and the training dataset via student-teacher learning. We evaluate our method on diverse CT, MRI, and microscopic cell datasets. Our framework significantly reduces user annotation time by a factor of 6.72×. Compared to other interactive segmentation methods, we reduce the time by a factor of 2.64×. Our method also outperforms prompting foundation models for segmentation by achieving a dice score of 0.9 in 3-4 interactions compared to 5-8 user interactions for the foundation model, significantly reducing annotation time for the CT and MRI volumes.

Abstract

This research utilises the capabilities of Google Earth Engine, a cloud-based computing platform, to conduct a comprehensive spatiotemporal assessment of the Coringa mangrove. Situated within the Coringa Wildlife Sanctuary in Andhra Pradesh, India, this mangrove – the country's second largest – is evaluated across various timeframes: 1999, 2002, 2007, 2013, 2017, and 2022. ML algorithms – Random Forest and Support Vector Machine with an RBF kernel – are employed. These algorithms analyze band composites derived from Landsat-7 ETM+ (1999, 2002, 2007), Landsat-8 (2013, 2017), and Sentinel-2 MSI (2022) satellite images. This analysis incorporates key spectral indices and utilises two spectral index thresholds for mangrove classification: one derived from established literature that identifies common thresholds at which mangroves typically occur (standard threshold), and the other is a customised threshold obtained from individual spectral index maps, tailored specifically to delineate mangrove areas in the study area (customized threshold). The results show that RF, particularly with the CT, outperforms all other methods, including RF with the ST and SVM with both thresholds, in terms of training and testing accuracy. These findings affirm the effectiveness of RF with the CT approach in accurately differentiating mangrove areas, emphasizing the critical role of threshold selection in enhancing the accuracy and competence of classification methods for mapping mangrove ecosystems.

Abstract

The significant improvements in point cloud representation learning have increased its applicability in many real-life applications, resulting in the need for lightweight, better-performing models. One widely proposed efficient method is knowledge distillation, where a lightweight model uses knowledge from large models. Very few works exist on distilling the knowledge for point clouds. Most of the work focuses on cross-modal-based approaches that make the method expensive to train. This paper proposes PointKAD, an adversarial knowledge distillation framework for point cloud-based tasks. PointKAD includes adversarial feature distillation and response distillation with the help of discriminators to extract and distill the representation of feature maps and logits. We conduct extensive experimental studies on both synthetic (ModelNet40) and real (ScanObjectNN) datasets to show that PointKAD achieves state-of-the-art results compared to the existing knowledge distillation methods for point cloud classification. Additionally, we present results on the part segmentation task, highlighting the efficacy of the PointKAD framework. Our experiments further reveal that PointKAD is capable of transferring knowledge across different tasks and datasets, showcasing its versatility. Furthermore, we demonstrate that PointKAD can be applied to a cross-modal training setup, achieving competitive performance with cross-modal-based point cloud methods for classification.

Abstract

Tornado Cash is a decentralized application (dApp) that runs on Ethereum Virtual Machine (EVM) compatible networks to enhance users’ privacy in terms of user transaction history over the blockchain. This dApp achieves its goal by enabling users to deposit currencies into designated pools and subsequently withdraw them, severing the link between depositor and withdrawer addresses. At deposit time, Tornado Cash communicates to users the level of privacy they will benefit from (anonymity set) by depositing currencies into one of its pools. Existing analyses have indicated discrepancies between the claimed anonymity set and the actual level of privacy provided, primarily attributed to users’ incorrect utilization of the dApp. This paper explores the road towards a new way to challenge the dApp’s proposed anonymity set by examining wallet fingerprints, a factor not directly related to user behavior within the application. The findings of this research shed light on the potential for creating links between clusters of users in TC according to the new proposed approach and raise a privacy concern within the Ethereum network, resulting in 13203 transactions, over 66948, linkable to the wallet used to initialize them.

Abstract

Moving Target Defense (MTD) has emerged as a proactive and dynamic framework to counteract evolving cyber threats. Traditional MTD approaches often rely on assumptions about the attacker’s knowledge and behavior. However, real-world scenarios are inherently more complex, with adaptive attackers and limited prior knowledge of their payoffs and intentions. This paper introduces a novel approach to MTD using a Markov Decision Process (MDP) model that does not rely on predefined attacker payoffs. Our framework integrates the attacker’s real-time responses into the defender’s MDP using a dynamic Bayesian Network. By employing a factored MDP model, we provide a comprehensive and realistic representation of the system and its dependencies. We also incorporate incremental updates to an attack response predictor as new data emerges, ensuring an adaptive and robust defense mechanism. Additionally, we consider the costs of switching configurations in MTD, integrating them into the reward structure to balance execution and defense costs. We first highlight the challenges of the problem through a theoretical negative result on regret. However, our empirical evaluations, conducted in network and web application settings, demonstrate the framework’s effectiveness in environments characterized by high uncertainty and dynamically changing attack surfaces.

Abstract

Text driven diffusion models have shown remarkable capabilities in editing images. However, when editing 3D scenes, existing works mostly rely on training a NeRF for 3D editing. Recent NeRF editing methods leverages edit operations by deploying 2D diffusion models and project these edits into 3D space. They require strong positional priors alongside text prompt to identify the edit location. These methods are operational on small 3D scenes and are more generalized to particular scene. They require training for each specific edit and cannot be exploited in real-time edits. To address these limitations, we propose a novel method, FreeEdit, to make edits in training free manner using mesh representations as a substitute for NeRF. Training-free methods are now a possibility because of the advances in foundation model’s space. We leverage these models to bring a training-free alternative and introduce solutions for insertion, replacement and deletion. We consider insertion, replacement and deletion as basic blocks for performing intricate edits with certain combinations of these operations. Given a text prompt and a 3D scene, our model is capable of identifying what object should be inserted/replaced or deleted and location where edit should be performed. We also introduce a novel algorithm as part of FreeEdit to find the optimal location on grounding object for placement. We evaluate our model by comparing it with baseline models on a wide range of scenes using quantitative and qualitative metrics and showcase the merits of our method with respect to others.