Abstract

We propose a second order gradient based method with ADAM and RMSprop for the training of generative adversarial networks. The proposed method is fastest to obtain similar accuracy when compared to prominent second order methods. Unlike state-of-the-art recent methods, it does not require solving a linear system, or it does not require additional mixed second derivative terms. We derive the fixed point iteration corresponding to proposed method, and show that the proposed method is convergent. The proposed method produces better or comparable inception scores, and comparable quality of images compared to other recently proposed state-of-the-art second order methods. Compared to first order methods such as ADAM, it produces significantly better inception scores. The proposed method is compared and validated on popular datasets such as FFHQ, LSUN, CIFAR10, MNIST, and Fashion MNIST for image generation tasks1 . Codes: https://github.com/misterpawan/acom

Abstract

Recent approaches to the tensor completion problem have often overlooked the nonnegative structure of the data. We consider the problem of learning a nonnegative lowrank tensor, and using duality theory, we propose a novel factorization of such tensors. The factorization decouples the nonnegative constraints from the low-rank constraints. The resulting problem is an optimization problem on manifolds, and we propose a variant of Riemannian conjugate gradients to solve it. We test the proposed algorithm across various tasks such as colour image inpainting, video completion, and hyperspectral image completion. Experimental results show that the proposed method outperforms many state-of-the-art tensor completion algorithms.

Abstract

Various studies have shown the advantages of using Machine Learning (ML) techniques for analog and digital IC design automation and optimization. Data scarcity is still an issue for electronic designs, while training highly accurate ML models. This work proposes generating and evaluating artificial data using generative adversarial networks (GANs) for circuit data to aid and improve the accuracy of ML models trained with a small training data set. The training data is obtained by various simulations in the Cadence Virtuoso, HSPICE, and Microcap design environment with TSMC 180nm and 22nm CMOS technology nodes. The artificial data is generated and tested for an appropriate set of analog and digital circuits. The experimental results show that the proposed artificial data generation significantly improves ML models and reduces the percentage error by more than 50% of the original percentage error, which were previously trained with insufficient data. Furthermore, this research aims to contribute to the extensive application of AI/ML in the field of VLSI design and technology by relieving the training data availability-related challenges.

Abstract

Extreme multilabel classification or XML, in short, has emerged as a new subtopic of interest in machine learning. Compared to traditional multilabel classification, here the number of labels is extremely large, hence the name extreme multilabel classification. Using classical one versus all classification wont scale in this case due to large number of labels, same is true for any other classifiers. Embedding of labels as well as features into smaller label space is an essential first step. Moreover, other issues include existance of head and tail labels, where tail labels are labels which exist in relatively smaller number of given samples. The existence of tail labels creates issues during embedding. This area has invited application of wide range of approaches ranging from bit compression motivated from compressed sensing, tree based embeddings, deep learning based latent space embedding including using attention weights, linear algebra based embeddings such as SVD, clustering, hashing, to name a few. The community has come up with a useful set of metrics to identify the correctly the prediction for head or tail labels. Keywords: extreme classification, head and tail labels, compressed sensing, deep learning, attention

Abstract

We consider the problem of learning tensors from partial observa- tions with structural constraints, under the low-rank assumption. To this end, we propose a general convex low-rank regularizer, pa- rameterized by linear maps, that extends the existing regularizers. Different choices of the linear maps lead to different convex low- rank regularizers. The resulting problem is called the generalized structured low-rank tensor learning problem. To solve this problem, we use duality theory to reformulate it into simpler sub-problems. The dual problem contains a rich geometric structure, which we exploit to develop first-order and second-order Riemannian opti- mization algorithms. The associated duality gap is derived, and it is shown to be zero. Moreover, we experimentally verify the cor- rectness of our algorithm on several special cases of the proposed general framework.

Abstract

In this paper, we study the min-max optimization problems on Riemannian manifolds. We introduce a Riemannian Hamiltonian function, minimization of which serves as a proxy for solving the original min-max problems. Under the Riemannian Polyak–Łojasiewicz (PL) condition on the Hamiltonian function, its minimizer corresponds to the desired min-max saddle point. We also provide cases where this condition is satisfied. To minimize the Hamiltonian function, we propose Riemannian Hamiltonian methods (RHM) and present their convergence analysis. We extend RHM to include consensus regularization and to the stochastic setting. We illustrate the efficacy of the proposed RHM in applications such as subspace robust Wasserstein distance, robust training of neural networks, and generative adversarial networks.

Abstract

Transformers, which were introduced for solving the task of machine translation, have expanded their utility in multiple domains. A recent application of transformers is in solving elementary mathematics problems. In this paper, we use a hybrid tokenization technique for encoding the mathematics and science problems and answers, which is used to train the transformer. We compare the performance of our tokenization with that of the char-to-char tokenzation in solving various types of mathematics and science problems. We discuss the accuracy, memory usage, and time to train the model with proposed tokenization. The proposed tokenization shows higher accuracy for some problems, and requires lesser memory compared to char-to-char tokenization. We propose an extended dataset of science and mathematics problems that consists of billions of samples in questionanswer format in raw text. Code and Dataset: https: //github.com/misterpawan/scimat2

Abstract

We propose a novel Riemannian method called “RXML" for solving the Extreme multi-label classification problem that exploits the geometric structure of the sparse low-dimensional local embedding models. A constrained optimization problem is formulated as an optimization problem on a matrix manifold and solved using a Riemannian optimization method. A proof of convergence for the proposed Riemannian optimization method is stated. The proposed approach is tested on several real-world large scale multi-label datasets, and its usefulness is demonstrated through numerical experiments. Experimental results show that RXML improves the trade-off between train time and accuracy. At the similar level of accuracy, the train time of RXML was 1.5 to 4 times faster than that of AnnexML and EXMLDS-4, which are the state-of-the-art embedding- based methods.

Abstract

Datasets play an important role in driving innovation in algorithms and architectures for supervised deep learning tasks. Numerous datasets exist for images, language translation, etc. One of the interesting challenge problems for deep learning is to solve high school problems in mathematics and sciences. To this end, a comprehensive set of dataset containing hundreds of millions of samples, and the generation modules is required that can propel research for these problems. In this paper, a large set of datasets covering mathematics and science problems is proposed, and the dataset generation codes are proposed. Test results on the proposed datasets for character-to-character transformer architecture show promising results with test accuracy above 95%, however, for some datasets it shows test accuracy of below 30%

Abstract

A fast, robust, parallel, and parameter free version of a frequency filtering preconditioner is proposed for linear systems corresponding to diffusion equation on a structured grid. Proposed solver is faster than the state-of-the-art solvers