Abstract

The rapid growth of deep neural network models in the face domain has led to their adoption in safety-critical applications. However, a crucial limitation hindering their widespread deployment is the lack of comprehensive understanding of how these models work and the inability to explain their de- cisions. Explainability is essential for ensuring the correctness, reliability, and fairness of AI systems, and there is a growing recognition of its importance across AI applications. Despite the significance of explainability, most current methods are designed for general object recognition tasks and cannot be directly applied to the face domain. Faces are highly structured objects, and face tasks often in- volve fine-grained details, making them unique and distinct from general object recognition. This thesis aims to bridge the gap in explainability literature for the face domain by providing novel methods for interpreting and analyzing deep face representations. In this thesis, we embark on a comprehensive journey of interpreting and analyzing deep face repre- sentations to uncover the underlying mechanisms behind DNN-based face-processing models. We first visualize face representations and introduce methods to identify functional concepts in face representa- tions using ’cross-task aware filters’ (CRAFT). Our approach includes an efficient task-aware pruning method using CRAFTs. We also present state-of-the-art Canonical Saliency Maps (CMS) to pinpoint critical input features. We thoroughly analyze deep face representations to understand the learned fea- tures and their functional relevance in different face tasks. To further enhance our understanding of human attention in the context of driving behavior, we investigate driver gaze patterns and develop DashGaze, a large-scale naturalistic driver gaze dataset . Using this dataset, we propose an innovative calibration-free driver gaze estimation algorithm that provides valuable information for studying and predicting driver behavior. The comprehensive overview, experimental studies, and analyses presented in this thesis contribute to the wider adoption of explainability methods in face-processing tasks, enabling safer and more trust- worthy deployment of deep-face algorithms in real-world applications. By shedding light on the inner workings of these models and their biases, this work paves the way for the responsible and ethical development of AI technologies in the face domain.

Abstract

The field of therapeutic peptide design is ripe for transformation, fueled by the convergence of biotechnology and artificial intelligence. Peptides, short chains of amino acids, offer a promising avenue for targeted drug therapies due to their inherent advantages over small molecules, including specificity and reduced side effects. However, the development of peptide therapeutics has been hindered by their limited oral bioavailability and susceptibility to enzymatic degradation. Recent advancements in deep learning techniques have opened new possibilities for addressing these challenges through innovative peptide design strategies. This thesis explores the development of a novel hybrid deep learning framework for de novo peptide design. By harnessing the power of diffusion models, known for their ability to learn complex data distributions, and integrating them with binding affinity maximization algorithms, we have created a system capable of generating peptide sequences optimized for specific target receptors. To demonstrate the applicability of this framework, we focus on designing therapeutic peptides targeting proteins ex- pressed by Plasmodium falciparum Erythrocyte Membrane Protein 1 (PfEMP1) genes, key contributors to malaria pathogenesis. Our results highlight the potential of this hybrid deep learning approach to revolutionize peptide drug discovery. By generating peptide candidates conditioned on the binding sites of target receptors, we offer a promising avenue for developing effective therapies for malaria and other diseases. This research underscores the transformative power of AI in peptide therapeutics, paving the way for a new era of precision medicine with enhanced efficacy and reduced toxicity.

Abstract

Geospatial data capture and handling of indoor spaces is increasing over the years and has had a varied history of data sources ranging from architectural and building drawings to indoor data acquisition approaches. While these have been more data format and information driven primarily for the physical representation of spaces, it is important to note that many application require for the semantic information to be made available for enriching their usecases. This work proposes a space classification model leading to an ontology for indoor spaces that accounts for both the semantic and geometric characteristics of the spaces. Furthermore, a Space semantic model is defined, based on this ontology, which can form the basis for extended data models which can be used appropriately in multiple applications. To demonstrate the utility of the model, an extension to the IndoorGML data standard is presented with a set of proposed classes that can help capture both the syntactic and semantic components of the model. It is expected that these proposed classes can be appropriately harnessed for use in diverse applications ranging from indoor data visualization to more user customised building evacuation path planning with a semantic overtone. The aim of the work presented here is to showcase the benefits of semantically rich indoor data and propose a model to do so. This work hopes to lay a foundation and pave way for more sophisticated and application/domain targeted data models which incorporates both semantic and syntactic properties of indoor space.

Abstract

Non-coding RNAs, including non-coding regions of pre-mRNAs, though they do not code for protein, play key functional and regulatory roles in gene expression at the transcriptional and post-transcriptional stages, influencing a wide range of biological processes. Their huge impact on several molecular mechanisms underpins their contribution to the development and progression of a broad spectrum of diseases, including cancer. With the availability of RNA structurome data and that of advanced RNA folding algorithms, it is possible to study the structural and conformational space spanned by these regulatory RNAs. These studies, in conjunction with detailed investigation of the associated regulatory processes and of genetic diseaserelated variations thereof, could help us in exploring new approaches towards understanding, diagnosing, and managing genetic diseases. We have explored disease-causing RNA structural disruptions called riboSNitches, in the splice junctions of pre-mRNAs, and shortlisted potential riboSNitches associated with somatic cancer mutations, aberrant splice mutations, and genetic diseases. We have developed an approach that is more reliable than other available methods for riboSNitch detection. Circular RNAs (circRNAs), one of the largest classes of non-coding RNAs, have exhibited a regulatory role in gene expression by acting as miRNA sponges. A significant number of studies have explored their capability as therapeutic agents and as biomarkers in several pathological conditions associated with their unique characteristics, such as diversity, stability, and abundance of tissue-specific expression. It was proposed that non-coding RNA transcripts, especially circRNAs and long noncoding RNAs (lncRNAs), form an extensive network by competitively binding to miRNA binding sites. This network, termed ceRNA/competitive endogenous network (circRNA/lncRNA-miRNA-mRNA), was demonstrated to regulate the translation of miRNA target genes. In this study, we proposed ceRNA network construction based on atrial fibrillation, one of the widely studied diseases for ceRNA networks, by exploring the existing approaches. Our review process led to the compilation of results highlighting promising candidate biomarkers for atrial fibrillation. We leveraged our comprehensive ceRNA network construction approach to investigate ceRNA networks in obesity, a disease with limited exploration in the context of ceRNA networks. To elucidate the role of non-coding RNAs in obesity, we constructed a visceral adipose tissue-specific integrated circRNA-miRNA-mRNA network, identified potential miRNA targets that regulate the mechanisms of obesity, and cataloged the circRNAs and miRNAs involved in the molecular pathways contributing to the disease. We also identified the candidate genes that potentially link to obesity-driven type 2 diabetes. To unravel the connections between obesity, diabetes, and cancer, we extended our analysis to pan-cancer data by exploring genes with similar expression patterns in obesity and diabetes. Our findings report associations between these genes and various cancer types that require further exploration to understand their underlying mechanisms. Taken together, our studies on non-coding circular RNAs and structure-disrupting mutations in the splice junctions of pre-mRNAs that regulate gene expression a) provide insights into the gene-specific and disease-specific mechanisms and b) identify potential biomarkers and regulatory RNA structures that can be explored for RNA-based therapies

Abstract

Coal has been one of the most important fossil fuels for power generation since the Industrial Revolution and has driven many nations towards industrial development. In the immediate post-independence years, the geological abundance of coal in India was framed as a boon to indigenous industries and their development; coal was seen as an important raw material that would ensure the newly independent nation’s industrial self-reliance. Coal, in the form of thermal power generation, has fueled various critical sectors of the country, ranging from transportation to manufacturing, and has contributed to the economic growth of the nation. This interconnectedness also underscores the inseparability of coal and power generation. Over time, there has been an exponential surge in domestic consumption of power, from fueling industries to lighting up of homes, leading to an increase in demand for coal. Though there has been a surge of interest and viability of alternative greener energy production in India and globally through renewable sources, coal is still the go-to raw material, accounting for 73% of the country’s energy generation and 44% of the total energy consumed in 2021 [1]. Similarly, there has been a multifold increase in the per-capita electric power consumption in recent times, at 637 kWh in 2010 [2], which is one-fifth the global average, to 1255 kWh in 2021 [3], which is one-third of the global average. The burgeoning demand from both industries and public use of electricity has driven an increase in power generation, consequently increasing the demand for coal. While efforts are being made through the import of coal, there has also been a push to make the nation self-reliant. This has brought about significant reforms by the ministry, such as the opening up of new mines, mechanisation of existing mines, adoption of technologically advanced tools, and extractive open-pit mining, amongst other measures. These developments have had social, economic and environmental consequences. Like any other extractive industry, coal also carries with it a contentious reputation for disrupting the environment in which it occurs. Coal, its mining in particular, has an indisputable history of disrupting the social fabric, compromising public health, and disturbing the environmental equilibrium in its surroundings. These impacts can be profound and extend across generations leaving lasting scars on the landscape. The collateral damage caused by coal mining ranges from severe environmental issues such as air pollution, water contamination, and land degradation to social implications like displacement of indigenous communities, livelihood loss, and disruption of local landscape, all while fueling the development engine of the nation. This paradoxical nature of extractive mining can be described through the Resource Curse theory [4], wherein places rich in natural resources often experience economic challenges and political and social instability (Auty, 1993). This holds for coal mines around the world due to the similar nature of the political economy surrounding coal. In India, the issue is magnified owing to the recent push towards an increase in indigenous production combined with the fact that coal-rich areas are inhabited by forest dwellers and people directly or indirectly reliant on land for sustenance. Despite its disruptive nature, coal mining continues to persist and proliferate due to the economic imperative of employment provision (however declining) and economic contribution. The push for an increase in indigenous production of coal, which is aimed at reducing the import, has forced all the existing mines to expand at an unprecedented pace and has spurred the opening up of new mines as well. Therefore, the changes happening in the vicinity of mining sites are also happening swiftly, and there is, thus, a need to keep track and account of them. While the expansion and opening up of every mine is contingent upon obtaining environmental clearance as part of the Environmental (Protection) Act 1986, of which the environmental and social impact assessments are a part of [5], many argue that these assessments lack efficacy and are often unreliable (Lahiri-Dutt, Krishnan & Ahmad, 2014). Moreover, the on-ground changes happening after the operation of a mine, like socio-cultural changes, cannot be accounted for beforehand and have to be analysed later on. Therefore, research into the socio-economic and environmental impacts is necessary. When analysed in-depth, choosing a specific study area will allow for a comprehensive understanding of the socio-economic and environmental dynamics in the context of coal mining. This allows for a more nuanced understanding of the complexities involved, and by localising the context, a multidimensional, accurate impact assessment can be done [6]. A temporal study of the study area is also viable in case of context limitation, which would give

Abstract

Lupus Nephritis (LN), a severe manifestation of systemic lupus erythematosus (SLE), presents significant diagnostic challenges due to the complex nature of renal pathology. Traditional methods for LN classification are labour-intensive, requiring glomerular-level labelling of whole slide images (WSIs), which leads to high inter- and intra-observer variability, lengthy turnaround times, and difficulty in identifying molecular biomarkers and morphological features. These issues highlight the need for automated and efficient methods that can handle the immense size and complexity of WSIs. Similarly, traditional immunoscoring methods for assessing immune responses within colon tissues are also labour-intensive and prone to variability. These challenges highlight the need for automated and efficient diagnostic techniques that can handle the immense size and complexity of WSIs and provide accurate diagnosis and prognosis. This thesis addresses these challenges through two main contributions: the development of LupusNet, a novel deep-learning model for LN classification using slide-level labels, and the creation of an open-source web-based tool for the automatic calculation of Immunoscore. LupusNet leverages Multiple Instance Learning (MIL) and advanced attention mechanisms to classify LN subtypes without requiring detailed glomerular annotations. The proposed method demonstrates significant improvements in classification accuracy, achieving an AUC score of 91.0%, an F1 score of 77.3%, and an accuracy of 81.1% on a comprehensive dataset of multi-stained LN digital histopathology WSIs from the Indian population. In parallel, a web-based application for immunoscoring has been developed, allowing users to upload WSIs and annotations for Core of Tumor (CT) and Invasive Margin (IM). The integrated algorithm calculates the immunoscore for individual CT and IM regions, only quantifying the immune response within the annotated tissue region. This pipeline is designed to be user-friendly and accessible, facilitating its use in clinical and research settings. Our pipeline is an endto-end solution for calculating the Immunoscore, overcoming several limitations of already available tools. Extensive experiments and evaluations of both LupusNet and the immunoscoring pipeline highlight their effectiveness and potential for practical integration into clinical workflows. The findings underscore the importance of leveraging digital pathology and machine learning to improve diagnostic accuracy, reduce pathologist workload, and enhance the scalability of LN classification and immunoscoring applications. The contributions of this research provide valuable insights and tools for the field of medical image analysis, with implications for advancing the diagnosis and treatment of complex diseases such as LN and enhancing the overall process of immunotherapy treatments.

Abstract

The emergence of vehicle connectivity technologies and associated applications have paved the way for increased consumer interest in connected vehicles. With the rise in state-of-the-art communication modes for vehicles such as vehicle to vehicle (V2V), vehicle to infrastructure (V2I) and vehicle to cloud (V2C), modern vehicles are increasingly being connected to cloud and fog/edge nodes. These modern day vehicles are now capable of sending/receiving vast amounts of data and offloading computation (which is one possible service) to servers thereby improving safety, comfort, driving experience, etc. In the early stages of connectivity, all the data communication and computation offloading happened between the cloud server and the vehicles. However, this is not feasible in scenarios having strict timing requirements and bandwidth cost constraints. Vehicular Edge Computing (VEC) demonstrated an efficient way to tackle the above problem. In order to optimally utilize the resources of the edge servers for data/service delivery, an efficient edge resource allocation framework needs to be developed. Prior system level VEC optimization frameworks for edge resource allocation did not consider which vehicles are present in the edge coverage region around the same time (which we call vehicle overlaps), which resulted in overestimation of resources. In a VEC system, vehicles receive very important and large quantity of data from edge nodes, which is termed as data delivery. In addition, edge nodes can execute some services and send the results back to the vehicle, which is called service delivery. Fast and efficient edge resource allocation for data/service delivery is important in order to serve as many vehicles as possible in the VEC system. However, edge resource allocation is complex with large number of edges and vehicles, while also considering vehicle flow parameters. In Chapter 3, we propose Edge-Pairwise Optimal Data/Service Delivery (E-PODS), which is a fast and efficient heuristic for data/service delivery. Through experiments with synthetic and real vehicular traces, we demonstrate that E-PODS is considerably faster than the optimal approach, while making resource allocations that are close to optimal in terms of total edge bandwidth cost and number of serviced vehicles. In Chapter 4, we propose an optimization framework for edge resource allocation that minimizes the bandwidth cost of data/service delivery to connected vehicles while considering the traffic flow and overlaps in vehicle presence at edge coverage region. Then, we propose an efficient heuristic to deliver data/service delivery based on minimizing global edge bandwidth cost gradient under vehicle overlaps. We demonstrate the improvement in resource allocation considering overlaps and also using the proposed heuristic with synthetic and real world traffic data. In Chapter 5, we present a novel scheduling technique for roadside clouds in vehicular networks, addressing high mobility and resource limitations. The Oldest-Request-First (ORF) algorithm priori- tizes older requests and redirects those near RSU edges, reducing response times and VM migrations. Simulations show ORF outperforms traditional methods, enhancing resource management by serving more requests within delay constraints.

Abstract

Cancer remains one of the leading causes of mortality worldwide, characterized by the uncontrolled growth and spread of abnormal cells. Effective management of cancer, including brain tumors and breast cancer, relies on precise detection, classification, and grading to inform treatment strategies. Traditional histopathological methods, although effective, are labor-intensive and subject to inter-pathologist variability. The advent of digital and computational pathology, using whole slide imaging (WSI) and deep learning algorithms, has revolutionized cancer diagnosis and prognosis by enabling more accurate, consistent, and efficient analysis of histopathological images. Computational pathology integrates advanced computational techniques with digitized histopathology slides, providing detailed quantitative assessments that significantly enhance diagnostic precision. Deep learning has shown remarkable performance in image classification, segmentation, and detection, making it particularly effective for analyzing complex medical images. By automating routine pathology tasks, deep learning improves diagnostic accuracy and reduces the workload on pathologists, facilitating faster and more reliable diagnoses. This thesis aims to utilize these advancements to improve the automated diagnosis and classification of brain gliomas and breast cancer, thus enhancing patient care through more personalized and timely treatment interventions. In the first part of the thesis, we focus on brain gliomas, a diverse group of tumors arising from glial cells in the central nervous system. Accurate typing, subtyping, and grading of gliomas are critical for determining prognosis and guiding treatment. Using a multiple-instance-learning (MIL) framework, we employ self-supervised pre-trained feature extractors and feature aggregators to classify glioma subtypes, determine tumor grades, and predict key immunohistochemistry (IHC) biomarkers such as IDH, ATRX, TP53, and Ki-67 using H&E images. Our study establishes new performance benchmarks in glioma subtype classification across multiple datasets, including the in-house developed Indian Pathology Brain Dataset (IPD-Brain), which provides a valuable resource for existing research. Using a ResNet-50 pretrained on histopathology datasets for feature extraction, combined with the DoubleTier Feature Distillation (DTFD) feature aggregator, our approach achieves state-of-the-art AUCs of 88.08 ± 3.98 on IPD-Brain and 95.81 ± 1.78 on the TCGA-Brain dataset for three-way glioma subtype classification. This work also highlights a significant correlation between the model’s decision-making processes and the diagnostic reasoning of pathologists, underscoring its capability to mimic professional diagnostic procedures. The second part of the thesis addresses breast cancer, the most common malignancy among women. Effective treatment of breast cancer requires precise detection and classification, particularly of IHC biomarkers like HER2, ER, and PR, which are critical for identifying breast cancer subtypes and informing treatment decisions. Traditional IHC classification relies on pathologist expertise, making it labor-intensive and prone to high inter-pathologist variability. We curate the Indian Pathology Breast Dataset (IPD-Breast) and develop automated deep learning models for accurate IHC subtype classification, specifically focusing on an evolving three-way HER2 classification scheme for better prognosis while also extending to binary and four-way classifications. Our proposed end-to-end ConvNext approach which uses low resolution whole slide images outperforms existing methods. For the three-way HER2 classification, it achieved an AUC of 91.79 ± 2.14, an F1 score of 83.52 ± 2.69, and an accuracy of 83.56 ± 2.80, outperforming existing approaches by at least 5.35% in F1 score. This study demonstrates that simple yet effective deep learning approaches can significantly improve accuracy and reproducibility in breast cancer classification, supporting their integration into clinical workflows for better patient outcomes. In summary, this thesis makes a significant contribution to the field of computational pathology by bridging the gap between computational techniques and clinical applications. Our contributions include the development of extensive open-source datasets for brain and breast cancer, the creation of robust diagnostic models, and the offering of valuable guidance on effective computational approaches for medical image analysis. By enhancing diagnostic precision and efficiency, these advancements support personalized treatment planning and improve patient outcomes. The integration of these computational tools into routine clinical practice holds promise for significantly advancing the field of pathology.

Abstract

This work presents several contributions to video representation learning and related multimodal tasks, addressing key challenges in datasets, efficient model adaptation using less data, and compositional and fine-grained visual understanding. Despite the rapid growth of online lecture videos in the past several years, video-language research has primarily focused on instructional videos/movies, resulting in a scarcity of specialized datasets for educational lecture videos. To address this, we first introduce AVLectures, a large-scale dataset of STEM lecture videos. It consists of 86 courses with over 2,350 lectures covering various STEM subjects. Each course contains video lectures, transcripts, OCR for lecture frames, and optionally lecture notes, slides, assignments, and related educational content that can inspire a variety of tasks. Next, we propose a novel unsupervised temporal segmentation task to segment lecture videos into bite-sized topics. We show that multimodal cues can be effectively utilized to learn lecture-aware representations for this task, facilitating a richer analysis of educational content. Next, we address the inefficiency of adapting pre-trained models like CLIP to videos. Existing methods typically rely on large-scale, sparsely annotated video caption datasets, resulting in slow and data-intensive adaptation. We propose SRL-CLIP, a novel approach that leverages the rich, structured semantic information within Semantic Role Labels (SRLs) for highly efficient adaptation. We use VidSitu for adaptation as it provides dense SRL annotations that holistically represent the entire video. SRL-CLIP achieves comparable or superior performance on various video understanding benchmarks (zero-shot retrieval, situation recognition, dense video captioning, and localization) compared to stateof-the-art models that possess 4−8× more parameters and are post-pretrained on up to 4000× times more data. To further explore the models’ understanding of visual content, we introduce three novel benchmarks. First, VELOCITI evaluates the compositional reasoning abilities of video-language models, focusing on their ability to bind semantic concepts through time. Second, we introduce NDLB, a frame work aimed at improving fine-grained image captioning, which uses self-retrieval as a key component along with a new benchmark to check if the model can capture subtle visual distinctions. Finally, we introduce D3, a benchmark specifically designed to evaluate the fine-grained visual discrimination capabilities of MLLMs using self-retrieval, further pushing the boundaries of fine-grained visual understanding. These contributions, which include novel datasets, efficient training recipes, and insightful benchmarks, collectively advance the state of the art in multimodal and video representation learning.

Abstract

Air pollution, whether indoors or outdoors, presents substantial threats to human health. These threats arise from the presence of pollutants in the air, which can have adverse effects on respiratory and cardiovascular systems, among other health impacts. Indoor air pollution often results from activities such as cooking, heating, and the use of certain household products, while outdoor air pollution stems from industrial emissions, vehicle exhaust, and other external sources. This paper tackles these pressing concerns through a comprehensive approach involving two interlinked studies. The first study delves into the evaluation of four low-cost carbon di-oxide (CO2) sensors, specifically designed for monitoring indoor air pollution. These sensors are tested against a high quality reference device in a controlled indoor environment at IIIT-H, India. The findings emphasize the importance of local calibration, even for factory-calibrated sensors, to ensure accurate and reliable measurements. Various machine learning algorithms are employed for calibration, enhancing the sensors’ performance and usability. The second study extends the investigation to outdoor air pollution, focusing on a construction area situated approximately 300 meters from a residential apartment in Hyderabad, India. Utilizing Internet of Things (IoT)-based air pollution nodes, the research assesses pollution patterns across both indoor and outdoor environments in four adjacent blocks near the construction site. Parameters such as particulate matter 2.5 and 10 (PM2.5, PM10), CO2, volatile organic compound (VOC), temperature, and humidity are measured to provide a comprehensive understanding of the air quality. The study also explores the correlation between different pollutants and employs box plots for detailed analysis. The combined insights from these studies offer a holistic perspective on the impact of air pollution on both indoor and outdoor environments. This research contributes valuable data to the ongoing global efforts aimed at addressing air quality issues. It not only enhances our understanding of the performance of low-cost sensors for indoor air monitoring but also sheds light on the far-reaching effects of construction-related air pollution. Ultimately, the findings aim to inform policymakers, construction companies, and the public, facilitating the development of effective strategies to mitigate the health risks associated with air pollution