Abstract

Robustness towards adversarial attacks is a vital property for classifiers in several applications such as autonomous driving, medical diagnosis, etc. Also, in such scenarios, where the cost of misclassification is very high, knowing when to abstain from prediction becomes crucial. A natural question is which surrogates can be used to ensure learning in scenarios where the input points are adversarially perturbed and the classifier can abstain from prediction? This paper aims to characterize and design surrogates calibrated in enquote{Adversarial Robust Reject Option} setting. First, we propose an adversarial robust reject option loss $ell_{d}^{gamma}$ and analyze it for the hypothesis set of linear classifiers ($mathcal{H}_{textrm{lin}}$). Next, we provide a complete characterization result for any surrogate to be $(ell_{d}^{gamma},mathcal{H}_{textrm{lin}})$- calibrated. To demonstrate the difficulty in designing surrogates to $ell_{d}^{gamma}$, we show negative calibration results for convex surrogates and quasi-concave conditional risk cases (these gave positive calibration in adversarial setting without reject option). We also empirically argue that Shifted Double Ramp Loss (DRL) and Shifted Double Sigmoid Loss (DSL) satisfy the calibration conditions. Finally, we demonstrate the robustness of shifted DRL and shifted DSL against adversarial perturbations on a synthetically generated dataset.

Abstract

We propose an approach to train a fair cost-based abstain option classifier. Existing literature on fairness in classification with abstention is limited, covering only coverage-based abstention models. In coverage-based abstention models, the target coverage is decided before-hand and is kept the same for all the groups. In contrast, cost-based approaches introduce a cost for abstention, which can cause uneven abstention rates between different groups, leading to an unfair system. We extend the independence and separation fairness criteria to consider abstention. We provide a model-agnostic in-processing algorithm to incorporate these constraints in the models. We demonstrate the algorithm’s efficacy by experimenting with two different cost-based abstain option classifiers. Additionally, we explore mixing constraints from independence and separation criteria into one model, which is impossible in a binary classification task.

Abstract

Automating end-to-end Exploratory Data Analysis (AutoEDA) is a challenging open problem, often tackled through Reinforcement Learning (RL) by learning to predict a sequence of analysis operations (FILTER, GROUP, etc). Defining rewards for each operation is a challenging task and existing methods rely on various emph{interestingness measures} to craft reward functions to capture the importance of each operation. In this work, we argue that not all of the essential features of what makes an operation important can be accurately captured mathematically using rewards. We propose an AutoEDA model trained through imitation learning from expert EDA sessions, bypassing the need for manually defined interestingness measures. Our method, based on generative adversarial imitation learning (GAIL), generalizes well across datasets, even with limited expert data. We also introduce a novel approach for generating synthetic EDA demonstrations for training. Our method outperforms the existing state-of-the-art end-to-end EDA approach on benchmarks by upto 3x, showing strong performance and generalization, while naturally capturing diverse interestingness measures in generated EDA sessions.

Abstract

This paper proposes deep architectures for learning instance-specific abstain (reject) option multiclass classifiers. The proposed approach uses novel bounded multiclass abstention loss for multiclass classification as a performance measure. This approach uses rejection cost as the rejection parameter in con- trast to coverage-based approaches. To show the effectiveness of the proposed approach, we experiment with several real-world datasets and compare them with state-of-the-art coverage-based and cost-of-rejection-based techniques. The ex- perimental results show that the proposed method improves performance over the state-of-the-art approaches.

Abstract

Deep learning systems have shown state-of-the-art performance in clinical prediction tasks. However, current research suggests that cleverly produced hostile images can trick these systems. Deep learning-based medical image classification algorithms have been questioned regarding their practical deployment. To address this problem, we provide an unsupervised learning technique for detecting adversarial attacks on medical images. Without identifying the attackers or reducing classification performance, our suggested strategy FNS (Features Normalization and Standardization), can detect adversarial attacks more effectively than earlier methods.

Abstract

We present a novel second-order cone programming frame- work to learn robust classifiers that can tolerate uncertainty in the obser- vations of partially labeled multiclass classification problems. We call it RPL-SVM. Our formulation is based on a chance-constrained framework. Experimental results show that RPL-SVM efficiently learns multiclass classifiers with missing values in a partial label setting.

Abstract

In this paper, we present online algorithm called {it Delaytron} for learning multi class classifiers using delayed bandit feedbacks. The sequence of feedback delays ${d_t}_{t=1}^T$ is unknown to the algorithm. At the $t$-th round, the algorithm observes an example $mathbf{x}_t$ and predicts a label $tilde{y}_t$ and receives the bandit feedback $mathbb{I}[tilde{y}_t=y_t]$ only $d_t$ rounds later. When $t+d_t>T$, we consider that the feedback for the $t$-th round is missing. We show that the proposed algorithm achieves regret of $mathcal{O}left(sqrt{frac{2 K}{gamma}left[frac{T}{2}+left(2+frac{L^2}{R^2Vert WVert_F^2}right)sum_{t=1}^Td_tright]}right)$ when the loss for each missing sample is upper bounded by $L$. In the case when the loss for missing samples is not upper bounded, the regret achieved by Delaytron is $mathcal{O}left(sqrt{frac{2 K}{gamma}left[frac{T}{2}+2sum_{t=1}^Td_t+vert mathcal{M}vert Tright]}right)$ where $mathcal{M}$ is the set of missing samples in $T$ rounds. These bounds were achieved with a constant step size which requires the knowledge of $T$ and $sum_{t=1}^Td_t$. For the case when $T$ and $sum_{t=1}^Td_t$ are unknown, we use a doubling trick for online learning and proposed Adaptive Delaytron. We show that Adaptive Delaytron achieves a regret bound of $mathcal{O}left(sqrt{T+sum_{t=1}^Td_t}right)$. We experimentally show that the proposed approach can learn efficient classifiers even with delayed bandit feedbacks and the accuracy does not degrade much due to delays in feedbacks.

Abstract

Online active learning algorithms reduce human labeling costs by querying only a subset of informative incoming instances from the data stream to update the classification model. Active learning for online multiclass classification under complete information has been well addressed; however, it remains unaddressed for the bandit setting. In this paper, we investigate online active learning techniques under the bandit feedback setting. We proposed an efficient algorithm for learning a multiclass classifier with bandit feedbacks under the active learning setting. The proposed algorithms enjoy a regret bound of the order O(logT) in the active learning setting as well as in the standard (non-active) bandit feedbacks. We show the effectiveness of the proposed approach using extensive experiments on several benchmark

Abstract

It is widely accepted that data preparation is one of the most timeconsuming steps of the machine learning (ML) lifecycle. It is also one of the most important steps, as the quality of data directly influences the quality of a model. In this tutorial, we will discuss the importance and the role of exploratory data analysis (EDA) and data visualisation techniques to find data quality issues and for data preparation, relevant to building ML pipelines. We will also discuss the latest advances in these fields and bring out areas that need innovation. To make the tutorial actionable for practitioners, we will also discuss the most popular open-source packages that one can get started with along with their strengths and weaknesses. Finally, we will discuss on the challenges posed by industry workloads and the gaps to be addressed to make data-centric AI real in industry settings

Abstract

No Results Found