Abstract

Image Attribution seeks to reveal the importance of image regions in the classifier’s final decision. Of the various ways to tackle this problem, the optimization based perspective is particularly intuitive: It applies the attribution as a mask on the image and reduces the attribution task to a loss, that can be optimized using gradient descent. Previous work has considered the goal as searching for the single best mask. Under this setup, however, there is a tendency towards trivial solutions of large masks with reduced discernment of the relative importances of regions. This has typically required auxiliary loss terms to control the area of the mask, however, their strength relative to the primary loss needs to be tuned. We challenge this necessity, by re-imagining attribution as an ordering of pixels according to importance. This ordering may be interpreted as a schedule which determines which locations get seen earlier and which later, allowing us to create a trajectory of masks from completely OFF to completely ON. We optimize through this sequence of masks of “all” areas and not just a single mask as in previous methods. We explore this setting, which we dub Saliency-as-Schedule (SaS), and demonstrate its effectiveness through experiments in a variety of settings, involving multiple datasets and CNN architectures. Further, we also propose a novel attribution task, feature saliency, where we use SaS to rate the influence of image regions on the intermediate feature maps of a CNN, and not just the class logit. Our findings suggest that SaS is a promising direction for the attribution problem. Our code will be available at https://github.com/tumbleweed/SaliencyasSchedule

Abstract

Image Attribution is the task of ascribing importance to regions of the input, for the final decision of a classifier. Many methods for attribution exist, and a recent development has been to use the image at multiple scales to improve the performance of some lightweight attribution methods. These gains have been demonstrated for methods that require a single or multiple but independent forward passes through the network; however, they haven’t been explored in the context of optimization-based attribution (OA) methods. As compared to other techniques, like CAMs or axiomatic methods, OA is attractive as it lends a natural formulation of the task without the need for any heuristic rules. However, the iterative way of solving the optimization problem presents challenges to straightforward utilization of multiple scales. We investigate this scenario and develop a novel 2-step approach that first discovers promising areas across scales and locations from the input and then runs Optimization based Attribution on them. We find that while a naive incorporation of image crops is unsuccessful, this 2-stage pipeline leads to improvements in performance. We provide qualitative and quantitative evidence for this and investigate the reasons for the improvement via multiple ablation experiments.

Abstract

In facial recognition systems, quality extends beyond conventional perceptual quality to features incorporating identity information. Most facial image datasets embark on factors such as illumination and pose, making current systems robust enough to these factors with impressive recognition performance. Still, it is imperative to acknowledge that age variation and emotional similarity significantly influence identity. Variations in these features might significantly deceive the FR systems. These features also serve as easy channels for adversarial attacks on FR systems that alter facial features, such as morphing. Hence, making FR systems sensitive to the variations introduced over the range of these features is critical. We propose that the Unified Tri-Feature Quality Metric (U3FQ) be incorporated. This novel assessment framework integrates three critical elements: age variance, facial expression similarity, and congruence scores from state-of-the-art recognition models such as VGG-Face, ArcFace, FaceNet, and OpenFace. The weighting U3FQ utilizes an advanced learning paradigm, employing a Regression Network model for facial image quality assessment. U3FQ was rigorously evaluated against general IQA techniques—BRISQUE, BLINDS-II, RankIQA, and specialized FIQA methodologies like PFE, SER-FIQA, and SDD-FIQA. Results are backed up with qualitative analysis on the effectiveness of the generated quality scores through DET plots of FNMR on different age ranges, expression matches heat maps, and Expected Verification Rate (EVRC) curves on various datasets.

Abstract

We introduce GPNN-CAM, a novel method for CNN explanation, that bridges two distinct areas of computer vision: Image Attribution, which aims to explain a predictor by highlighting image regions it finds important, and Single Image Generation (SIG), that focuses on learning how to generate variations of a single sample.GPNN-CAM leverages samples generated by Generative Patch Nearest Neighbors (GPNN) into a Class Activation Map (CAM) flavored attribution scheme. Our findings reveal that the incorporation of these samples yields remarkably effective results, enabling GPNN-CAM to demonstrate superior performance across multiple classifier architectures, and datasets.

Abstract

Air quality estimation through sensor-based methods is widely used. Nevertheless, their frequent failures and maintenance challenges constrain the scalability of air pollution monitoring efforts. Recently, it has been demonstrated that air quality estimation can be done using image-based methods. These methods offer several advantages including ease of use, scalability, and low cost. However, the accuracy of these methods hinges significantly on the diversity and magnitude of the dataset utilized. The advancement of air quality estimation through image analysis has been limited due to the lack of available datasets. Addressing this gap, we present TRAQID - Traffic-Related Air Quality Image Dataset, a novel dataset capturing 26,678 front and rear images of traffic alongside co-located weather parameters, multiple levels of Particulate Matters (PM) and Air Quality Index (AQI) values. Spanning over multiple seasons, with over 70 hours of data collection in the twin cities of Hyderabad and Secunderabad, India, the TRAQID offers diverse day and night imagery amid unstructured traffic conditions, encompassing six AQI categories ranging from “Good” to “Severe”. State-of-the-art air quality estimation techniques, which were trained on a smaller and less-diverse dataset, showed poor results on the dataset presented in this paper. TRAQID models various uncertainty types, including seasonal changes, unstructured traffic patterns, and lighting conditions. The information from the two views (front and rear) of the traffic can be combined to improve the estimation performance in such challenging conditions. As such, the TRAQID serves as a benchmark for image-based air quality estimation tasks and AQI prediction, given its diversity and magnitude.

Abstract

Forensic sketch-to-mugshot matching is a challenging task in face recognition, primarily hindered by the scarcity of annotated forensic sketches and the modality gap between sketches and photographs. To address this, we propose CLIP4Sketch, a novel approach that leverages diffusion models to generate a large and diverse set of sketch images, which helps in enhancing the performance of face recognition systems in sketch-to-mugshot matching. Our method utilizes Denoising Diffusion Probabilistic Models (DDPMs) to generate sketches with explicit control over identity and style. We combine CLIP and Adaface embeddings of a reference mugshot, along with textual descriptions of style, as the conditions to the diffusion model. We demonstrate the efficacy of our approach by generating a comprehensive dataset of sketches corresponding to mugshots and training a face recognition model on our synthetic data. Our results show significant improvements in sketch-to-mugshot matching accuracy over training on an existing, limited amount of real face sketch data, validating the potential of diffusion models in enhancing the performance of face recognition systems across modalities. We also compare our dataset with datasets generated using GAN-based methods to show its superiority.

Abstract

Portrait sketching involves capturing identity specific attributes of a real face with abstract lines and shades. Unlike photo-realistic images, a good portrait sketch generation method needs selective attention to detail, making the problem challenging. This paper introduces textbf{Portrait Sketching StyleGAN (PS-StyleGAN)}, a style transfer approach tailored for portrait sketch synthesis. We leverage the semantic $W+$ latent space of StyleGAN to generate portrait sketches, allowing us to make meaningful edits, like pose and expression alterations, without compromising identity. To achieve this, we propose the use of Attentive Affine transform blocks in our architecture, and a training strategy that allows us to change StyleGAN's output without finetuning it. These blocks learn to modify style latent code by paying attention to both content and style latent features, allowing us to adapt the outputs of StyleGAN in an inversion-consistent manner. Our approach uses only a few paired examples ($sim 100$) to model a style and has a short training time. We demonstrate PS-StyleGAN's superiority over the current state-of-the-art methods on various datasets, qualitatively and quantitatively.

Abstract

In the field of biometric security, the quality assessment of fingerprint images is paramount for boosting the accuracy of fingerprint recognition systems. These systems are fundamental for the secure and efficient authentication and identification of individuals. Our research presents FRBQ (Fingerprint Recognition-Based Quality), an innovative quality metric designed to navigate the limitations of the NFIQ2 model. FRBQ exploits deep learning algorithms in a weakly supervised setting and utilizes matching scores from DeepPrint, a Fixed-Length Fingerprint Representation Model. Each score is paired with labels indicating the robustness of fingerprint image matches. However, in a fully referenced setting, these labels can be subjective, lacking a clear definition of what "image quality" inherently means. This weakly labeled approach strives to capture diverse perspectives on image quality, potentially making it a more encompassing metric. In comparison to NFIQ2, our research showcases the superior performance of the FRBQ model. It not only correlates better with recognition scores but also effectively evaluates challenging images that NFIQ2 struggles with. Validated by the esteemed FVC 2004 dataset, FRBQ proves its efficacy in fingerprint image quality assessment. This study underscores the transformative potential of AI in biometrics, emphasizing its capability to capture details that traditional methods might overlook. Our work stresses the critical role of precise quality assessment in the evolution of fingerprint recognition systems.

Abstract

Fingerprint recognition stands as a pivotal component of biometric technology, with diverse applications from identity verification to advanced search tools. In this paper, we propose a unique method for deriving robust fingerprint representations by leveraging enhancement-based pre-training. Building on the achievements of U- Net-based fingerprint enhancement, our method employs a specialized encoder to derive representations from fingerprint images in a self-supervised manner. We further refine these representations, aiming to enhance the verification capabilities. Our experimental results, tested on publicly available fingerprint datasets, reveal a marked improvement in verification performance against established self-supervised training techniques. Our findings not only highlight the effectiveness of our method but also pave the way for potential advancements. Crucially, our research indicates that it s feasible to extract meaningful fingerprint representations from de- graded images without relying on enhanced samples.

Abstract

The increasing prominence of fingerprint recognition as a biomet ric identifier has made it more vulnerable to presentation attacks, specifically non-distal attacks that exploit ridge and minutiae pat terns foundinnon-distalphalanges.Inthisstudy,wepresentpresen tation attacks through non-distal/toe prints and a state-of-the-art lightweight inverted residual network that excels at differentiating between distal and non-distal prints, providing unrivaled perfor mance in terms of accuracy, inference time, and false negative rate (FNR). Our proposed model surpasses other statistical machine learning methods, such as variable-margin SVM, and lightweight models like MobileNet v2, MobileNet v3, and ResNet18. We meticu lously evaluate our model using a diverse array of datasets, includ ing the NIST dataset, an in-house collected dataset, a toe dataset, a synthetic dataset generated by VeriFinger software, and a six-class dataset. To assess performance when only minutiae points are avail able, we develop analgorithmthat converts fingerprints to minutiae points and subsequently reconstructs fingerprints. Furthermore, we examine the ridge density of distal and non-distal prints across datasets, emphasizing their similarities and underscoring the need for advanced detection techniques. To the best of our knowledge, this study represents the first endeavor to propose a solution for presentation attack detection in non-distal phalanges. Our research demonstrate various challenges of presentation attacks, the effectiveness of our approach, which holds the potential to significantly influence the domain of finger print recognition and security. By sharing our dataset, model, and experimental details with the research community, we aim to foster further advancements in this crucial area. Upon publication, we will make our dataset and experimental details available alongside the paper