Abstract

Modern large language models (LLMs) place extraordinary pres-
sure on memory and compute budgets, making principled compres-
sion indispensable for both deployment and continued training.
We present Hierarchical Sparse Plus Low-Rank (HSS) compres-
sion, a two-stage scheme that (i) removes the largest-magnitude
weights into a sparse matrix 𝑆 and (ii) applies a recursive Hierar-
chically Sparse Separable (HSS) low-rank factorisation to the dense
residual matrix. A recursive rank-reducing strategy and a reverse
Cuthill-Mckee (RCM) permutation are introduced to align high
weights towards the diagonal with the block-diagonal hierarchy,
maximising off-diagonal compressibility (because they are touched
only once). HSS is hardware-friendly: its matrix-vector multiply
reduces to one sparse and a sequence of thin-matrix multiplications,
and can be trained end-to-end with standard optimisers.
Experiments on LLaMA-7B show that targeting only the self-
attention projections (1.6B parameters of 𝑄, 𝐾, and 𝑉 matrices out
of a total 7B parameters) suffices to yield large memory savings
while retaining comparable state-of-the-art perplexity scores on test
samples of the WikiText dataset. For example, with a 30% sparsity
budget and an outer rank of 512, sHSS-RCM achieves a perplexity of
1.64, outperforming dense baselines and classical sparse-plus-SVD
variants, while also achieving significant memory savings.

Abstract

In recent times, denoising diffusion probabilistic models (DPMs) have proven to show significant success in medical image generation and denoising, while also serving as powerful representation learners for downstream tasks such as segmentation. However, their effectiveness in segmentation is limited by the need for detailed pixel-wise annotations, which are expensive, time-consuming, and require expert knowledge--a significant bottleneck in real-world clinical applications. In order to mitigate this limitation of label-efficiency, we propose a fast and efficient model named FastTextDiff, a diffusion-based segmentation model that integrates medical text annotations to enhance semantic representations.
Our approach leverages ModernBERT, a transformer-based language model capable of processing long medical text sequences, to establish a strong connection between textual annotations and semantic meaning in medical imaging. ModernBERT can efficiently encode clinical knowledge for directing segmentation tasks since it has been trained on both MIMIC-III and MIMIC-IV. Label-efficient segmentation with enhanced performance is made possible by cross-modal attention processes, which enable smooth interaction between visual and textual modalities.
This study validates ModernBERT as a quick and scalable substitute for Clinical BioBERT in diffusion-based segmentation pipelines and demonstrates the promise of multi-modal techniques for medical image analysis. By replacing Clinical BioBERT with ModernBERT, our model benefits from Flash Attention 2 for memory-efficient training, an alternating attention mechanism for computational efficiency, and a training corpus of 2 trillion tokens, significantly improving text-based medical image segmentation. Our experiments demonstrate that FastTextDiff achieves better segmentation performance and faster training compared to traditional diffusion-based models. We release this trained model on Hugging Face and GitHub.

Abstract

Skip to content
IOP Science home
Accessibility Help
Search
Journals
Books
Publishing Support
Login
Engineering Research Express
Inclusive Publishing Trusted Science, find out more.


Purpose-led Publishing, find out more.



ACCEPTED MANUSCRIPT
A label-free sensing of creatinine using radio frequency-driven lab-on-chip (loc) system
Andleeb Zahra1, Swarnim Sinha2, Alimpan Modak3, Imran Siddiqui4, Azeemuddin Syed5, Prabhakar Bhimalapuram6, Tapan K. Sau7, Pawan Kumar8 and Zia Abbas9

Accepted Manuscript online 2 August 2024 • © 2024 IOP Publishing Ltd
What is an Accepted Manuscript?

DOI 10.1088/2631-8695/ad6ad5
DownloadAccepted Manuscript PDF
Download PDF
Article metrics
6 Total downloads
Submit
Submit to this Journal
Permissions
Get permission to re-use this article

Share this article
Article and author information
Abstract
This paper presents a promising avenue of Radio Frequency (RF) biosensors for sensitive and real-time monitoring of creatinine detection. Knowing creatinine levels in the human body is related to its possible association with renal, muscular, and thyroid dysfunction. The detection was performed using an Inter-Digitated Capacitor (IDC) made of copper (Cu) metal over an FR4 substrate. To demonstrate our methodology, we have chosen Phosphate Buffer (PB) as our solvent for making the creatinine solutions of different concentrations. Moreover, Assayed Chemistry Control (ACC), a reference control consisting of human serum-based solutions has been mixed with the different concentrations of creatinine in a ratio of 1:9 to spike the creatinine value in the ACC solution. The sensor has been designed using a High-Frequency Structure Simulator (HFSS) tool with an operating frequency of 2.53 GHz. Then the design is fabricated over the FR4 printed circuit board (PCB) and tested using a Vector Network Analyzer (VNA). However, the sensitive area of the IDC is introduced to grade 4 Whatman filter paper for the Sample Under Test (SUT) handling unit. The main advantage of using Whatman filter paper is that the uniform spreading of liquid reduces experimental error, and less volume is required for testing the sample. The principal idea implemented in the biosensor design is to track the shift in the operating frequency in the presence of different concentrations of creatinine mix in ACC solution with Phosphate Buffer (PB) solution as a reference.

Abstract

Various studies have shown the advantages of using Machine Learning (ML) techniques for analog and digital IC design automation and optimization. Data scarcity is still an issue for electronic designs, while training highly accurate ML models. This work proposes generating and evaluating artificial data using generative adversarial networks (GANs) for circuit data to aid and improve the accuracy of ML models trained with a small training data set. The training data is obtained by various simulations in the Cadence Virtuoso, HSPICE, and Microcap design environment with TSMC 180 nm and 22 nm CMOS technology nodes. The artificial data is generated and tested for an appropriate set of analog circuits and digital cells. The experimental results show that the proposed artificial data generation significantly improves ML models and reduces the percentage error by more than 50% of the original percentage error, which were previously trained with insufficient data. Furthermore, this research aims to contribute to the extensive application of AI/ML in the field of VLSI design and technology by relieving the training data availability-related challenges.

Abstract

No Results Found

Abstract

Generative AI has seen remarkable growth over the past few years, with diffusion models being state-of-the-art for image generation. This study investigates the use of diffusion models in artificial data generation for electronic circuits to enhance the accuracy of subsequent machine learning models in tasks such as performance assessment, design, and testing when training data is usually known to be very limited. We utilize simulations in the HSPICE design environment with 22nm CMOS technology nodes to obtain representative real training data for our proposed diffusion model. Our results demonstrate the close resemblance of synthetic data using diffusion models to real data. We validate the quality of generated data and demonstrate that data augmentation is certainly effective in the predictive analysis of VLSI design for digital circuits.

Abstract

Generative AI has seen remarkable growth over the past few years, with diffusion models being state-of-the-art for image generation. This study investigates the use of diffusion models in generating artificial data generation for electronic circuits for enhancing the accuracy of subsequent machine learning models in tasks such as performance assessment, design, and testing when training data is usually known to be very limited. We utilize simulations in the HSPICE design environment with 22nm CMOS technology nodes to obtain representative real training data for our proposed diffusion model. Our results demonstrate the close resemblance of synthetic data using diffusion model to real data. We validate the quality of generated data, and demonstrate that data augmentation certainly effective in predictive analysis of VLSI design for digital circuits.

Abstract

This work presents a thermoelectric vibration and buckling behaviour of a bidirectional functionally graded piezoelectric porous (BD-FGPP) plate resting on the elastic foundation with arbitrary boundary conditions. The variation of the material properties in the FGPP plate is bidirectional, ie, along longitudinal and thickness directions, using the modified power law distributions. Also, the BD-FGPP plate carries various porosity distributions, ie, even, uneven, and symmetric center types. The governing equation of the FGPP plate has been obtained through Hamilton's principle and solved through a higher-order finite element approach. The correctness and usefulness of the current technique are confirmed and compared with existing model outcomes available in the literature. A comprehensive parametric study has investigated the effect of the elastic foundations, porous exponent with various porosity distributions

Abstract

A reliable critic is central to on-policy actor-critic learning. But it becomes challenging to learn a reliable critic in a multi-agent sparse reward scenario due to two factors: 1) The joint action space grows exponentially with the number of agents 2) This, combined with the reward sparseness and environment noise, leads to large sample requirements for accurate learning. We show that regularising the critic with spectral normalization (SN) enables it to learn more robustly, even in multi-agent on-policy sparse reward scenarios. Our experiments show that the regularised critic is quickly able to learn from the sparse rewarding experience in the complex SMAC and RWARE domains. These findings highlight the importance of regularisation in the critic for stable learning.

Abstract

Recent advances in Reinforcement Learning (RL) have led to many exciting applications. These advancements have been driven by improvements in both algorithms and engineering, which have resulted in faster training of RL agents. We present marl-jax, a multi-agent reinforcement learning software package for training and evaluating social generalization of the agents. The package is designed for training a population of agents in multi-agent environments and evaluating their ability to generalize to diverse background agents. It is built on top of DeepMind's JAX ecosystem~\cite{deepmind2020jax} and leverages the RL ecosystem developed by DeepMind. Our framework marl-jax is capable of working in cooperative and competitive, simultaneous-acting environments with multiple agents. The package offers an intuitive and user-friendly command-line interface for training a population and evaluating its generalization capabilities. In conclusion, marl-jax provides a valuable resource for researchers interested in exploring social generalization in the context of MARL. The open-source code for marl-jax is available at: \href{this https URL}{this https URL}