Abstract

Cancer remains one of the leading causes of mortality worldwide, characterized by the uncontrolled growth and spread of abnormal cells. Effective management of cancer, including brain tumors and breast cancer, relies on precise detection, classification, and grading to inform treatment strategies. Traditional histopathological methods, although effective, are labor-intensive and subject to inter-pathologist variability. The advent of digital and computational pathology, using whole slide imaging (WSI) and deep learning algorithms, has revolutionized cancer diagnosis and prognosis by enabling more accurate, consistent, and efficient analysis of histopathological images. Computational pathology integrates advanced computational techniques with digitized histopathology slides, providing detailed quantitative assessments that significantly enhance diagnostic precision. Deep learning has shown remarkable performance in image classification, segmentation, and detection, making it particularly effective for analyzing complex medical images. By automating routine pathology tasks, deep learning improves diagnostic accuracy and reduces the workload on pathologists, facilitating faster and more reliable diagnoses. This thesis aims to utilize these advancements to improve the automated diagnosis and classification of brain gliomas and breast cancer, thus enhancing patient care through more personalized and timely treatment interventions. In the first part of the thesis, we focus on brain gliomas, a diverse group of tumors arising from glial cells in the central nervous system. Accurate typing, subtyping, and grading of gliomas are critical for determining prognosis and guiding treatment. Using a multiple-instance-learning (MIL) framework, we employ self-supervised pre-trained feature extractors and feature aggregators to classify glioma subtypes, determine tumor grades, and predict key immunohistochemistry (IHC) biomarkers such as IDH, ATRX, TP53, and Ki-67 using H&E images. Our study establishes new performance benchmarks in glioma subtype classification across multiple datasets, including the in-house developed Indian Pathology Brain Dataset (IPD-Brain), which provides a valuable resource for existing research. Using a ResNet-50 pretrained on histopathology datasets for feature extraction, combined with the DoubleTier Feature Distillation (DTFD) feature aggregator, our approach achieves state-of-the-art AUCs of 88.08 ± 3.98 on IPD-Brain and 95.81 ± 1.78 on the TCGA-Brain dataset for three-way glioma subtype classification. This work also highlights a significant correlation between the model’s decision-making processes and the diagnostic reasoning of pathologists, underscoring its capability to mimic professional diagnostic procedures. The second part of the thesis addresses breast cancer, the most common malignancy among women. Effective treatment of breast cancer requires precise detection and classification, particularly of IHC biomarkers like HER2, ER, and PR, which are critical for identifying breast cancer subtypes and informing treatment decisions. Traditional IHC classification relies on pathologist expertise, making it labor-intensive and prone to high inter-pathologist variability. We curate the Indian Pathology Breast Dataset (IPD-Breast) and develop automated deep learning models for accurate IHC subtype classification, specifically focusing on an evolving three-way HER2 classification scheme for better prognosis while also extending to binary and four-way classifications. Our proposed end-to-end ConvNext approach which uses low resolution whole slide images outperforms existing methods. For the three-way HER2 classification, it achieved an AUC of 91.79 ± 2.14, an F1 score of 83.52 ± 2.69, and an accuracy of 83.56 ± 2.80, outperforming existing approaches by at least 5.35% in F1 score. This study demonstrates that simple yet effective deep learning approaches can significantly improve accuracy and reproducibility in breast cancer classification, supporting their integration into clinical workflows for better patient outcomes. In summary, this thesis makes a significant contribution to the field of computational pathology by bridging the gap between computational techniques and clinical applications. Our contributions include the development of extensive open-source datasets for brain and breast cancer, the creation of robust diagnostic models, and the offering of valuable guidance on effective computational approaches for medical image analysis. By enhancing diagnostic precision and efficiency, these advancements support personalized treatment planning and improve patient outcomes. The integration of these computational tools into routine clinical practice holds promise for significantly advancing the field of pathology.

Abstract

This work presents several contributions to video representation learning and related multimodal tasks, addressing key challenges in datasets, efficient model adaptation using less data, and compositional and fine-grained visual understanding. Despite the rapid growth of online lecture videos in the past several years, video-language research has primarily focused on instructional videos/movies, resulting in a scarcity of specialized datasets for educational lecture videos. To address this, we first introduce AVLectures, a large-scale dataset of STEM lecture videos. It consists of 86 courses with over 2,350 lectures covering various STEM subjects. Each course contains video lectures, transcripts, OCR for lecture frames, and optionally lecture notes, slides, assignments, and related educational content that can inspire a variety of tasks. Next, we propose a novel unsupervised temporal segmentation task to segment lecture videos into bite-sized topics. We show that multimodal cues can be effectively utilized to learn lecture-aware representations for this task, facilitating a richer analysis of educational content. Next, we address the inefficiency of adapting pre-trained models like CLIP to videos. Existing methods typically rely on large-scale, sparsely annotated video caption datasets, resulting in slow and data-intensive adaptation. We propose SRL-CLIP, a novel approach that leverages the rich, structured semantic information within Semantic Role Labels (SRLs) for highly efficient adaptation. We use VidSitu for adaptation as it provides dense SRL annotations that holistically represent the entire video. SRL-CLIP achieves comparable or superior performance on various video understanding benchmarks (zero-shot retrieval, situation recognition, dense video captioning, and localization) compared to stateof-the-art models that possess 4−8× more parameters and are post-pretrained on up to 4000× times more data. To further explore the models’ understanding of visual content, we introduce three novel benchmarks. First, VELOCITI evaluates the compositional reasoning abilities of video-language models, focusing on their ability to bind semantic concepts through time. Second, we introduce NDLB, a frame work aimed at improving fine-grained image captioning, which uses self-retrieval as a key component along with a new benchmark to check if the model can capture subtle visual distinctions. Finally, we introduce D3, a benchmark specifically designed to evaluate the fine-grained visual discrimination capabilities of MLLMs using self-retrieval, further pushing the boundaries of fine-grained visual understanding. These contributions, which include novel datasets, efficient training recipes, and insightful benchmarks, collectively advance the state of the art in multimodal and video representation learning.

Abstract

Machine Translation (MT), the automated process of translating text or speech between languages using computational linguistic algorithms, has drastically transformed global communication. By breaking down language barriers, MT has enabled seamless collaboration across different social, cultural, and geographical backgrounds. Although several decades of research have produced state-of-the-art models like Google Translate, these systems often falter when translating domain-specific content, such as specialized academic texts or technical manuals. These models, trained primarily on large, general-purpose datasets, struggle to accurately capture domain-specific vocabulary and linguistic nuances. For example, terms like ”chromosome” in a microbiology textbook are often poorly translated by these systems. In this thesis, we propose DomainTrans, a MT model designed to enhance translation accuracy for domain-specific texts while maintaining performance on general content. The core idea here is to form a bijection between multi-domain translation and multi-lingual translation. By treating different domain datasets as analogous to distinct languages, we leverage multi-lingual translation techniques to develop a more versatile and adaptable MT model. This thesis also studies and experiments with other key peripherals around this topic. First being domain classification, where we identify the domain to which a given text belongs as a precursor to translation. We particularly lay special emphasis on fine grained domain classification. Next we look at the effect of data availability while fine tuning models for machine translation. Towards end end of the thesis we also explore Knowledge Distillation - the process of condensing the learning done by a large or a set of large models onto a much smaller efficient to deploy model while trying to maintain similar accuracy.

Abstract

This thesis explores the intricate relationship among color saturation, salience, and visual perception, with a focus on defining color salience and investigating the potential for universal color salience. The research delves into how color saturation—defined as the intensity and purity of color—affects visual perception and contributes to the noticeability of objects in their surroundings. Special attention is given to addressing the limitations of existing theories concerning individual differences in color perception, particularly in scenarios involving color ambiguous stimuli like the internet phenomenon #thedress, where perceptions varied widely. The study draws inspiration from diverse sources, including a saree featuring 50,000 colors, prompting an examination of perceptual limits in color vision and the impact of psychophysical properties on color dominance within ensembles. Through experiments involving tasks to select dominant colors and eye-tracking analyses, the research investigates factors influencing perceived color dominance, such as hue, saturation, and value. The role of individual differences in color perception is explored using clustering techniques to group observers based on their color choices, revealing distinct perceptual strategies and trends within these clusters. Key findings challenge previous assumptions, particularly regarding the salience of hues like blue, which are found to exhibit significant dominance effects contrary to previous research. The study introduces the Relative Perceived Hue Difference (RPHD) metric to quantify color dominance, showing instances where perceived dominance does not align with actual color distribution in stimuli. Eye-tracking data further suggests that sustained attention, reflected in largest fixation points, may be influenced by dominant hues in binary stimuli, highlighting complexities in dominant color perception beyond initial visual capture. This research contributes to advancing our understanding of color perception, salience, and the underlying factors driving visual attention in individuals, offering insights into the intricate dynamics of how we perceive and prioritize colors in our visual environment.

Abstract

This research explores Bayesian methods for predicting model uncertainty in remote sensing data, focusing on two distinct approaches: Bayesian linear regression and a novel Bayesian Nonlinear Neural Network (BN3) model. Both methods utilize SENTINEL-2 Normalized Difference Vegetation Index (NDVI) data from agricultural fields in Uttar Pradesh and Madhya Pradesh, India, for time series predictions. Bayesian linear regression, a probabilistic deep learning approach, evaluates model uncertainty through prior and posterior probability parameters using Bayesian inference. The analysis reveals high model uncertainty in predicted NDVI values, which diminishes as the data volume increases. This reduction in uncertainty is accompanied by sharper posterior density, indicating reduced variance. Regression analysis with Gaussian distribution further validates this trend, confirming that greater data availability leads to improved model certainty. Complementing this approach, the study develops the BN3 model to address model uncertainty in neural networks. BN3 employs probabilistic modeling with variational hidden layers and output layers characterized by Gaussian distributions for variational posteriors. It assesses model uncertainty using dense variational layers with continuous, differentiable nonlinear activation functions and varying degrees of freedom. The study evaluates the goodness of fit for nonlinear models, including polynomials and neural networks, using RMSE as a loss function. Comparison with Deep Ensemble methods highlights the BN3 models effectiveness, particularly in reducing uncertainty for cubic models. Visual analyses demonstrate BN3s consistent performance in minimizing model uncertainty across datasets, even with varying data availability. These findings demonstrate BN3s potential as a robust tool for improving uncertainty quantification in decision support systems.

Abstract

The widespread use of artificial intelligence has resulted in significant advancements in the field of Natural Language Processing, with text summarization and question answering models being some of the most sought-after applications. By understanding the Causal Relationships between Events within a text, we can gain valuable insights into how Events influence each other, leading to more accurate and sophisticated model outcomes. This thesis is an attempt towards improving Event Causality through various ways. The thesis commences with a comprehensive examination of Events and States, outlining their various classifications and the relationships between them. This foundational discussion establishes the context for the exploration of Event Causality. It then delves into existing research on Event Causality identification highlighting areas for potential contribution Recognizing the scarcity of datasets focused on Event Causality in the Hindi language, this thesis introduces the Hindi Causal TimeBank. This dataset is built upon the existing Hindi TimeBank containing about 1000 news articles and can serve as a crucial resource for developing and training Event Causality models for Hindi-language text. The thesis elaborates on the different types of Causal Relations annotated within the dataset. The primary focus of this thesis lies in the advancement of Event Causality identification. Specifically, it explores the use of graph-based models (Graph Attention Networks (GAT)). GATs are well-suited for representing complex relationships between entities and Events. This thesis presents a series of experiments designed to improve results on Event Causality identification, aiming to outperform the current state-of-the-art methods. The performance of these models is evaluated across a range of datasets, including the newly developed Hindi Causal TimeBank. This thesis makes a significant contribution to the field of Event Causality. Improved Event Causality understanding has implications for information retrieval, machine translation, and knowledge graph construction. The introduction of the Hindi Causal TimeBank further expands the scope of this research into a crucial and understudied language.

Abstract

Enhancing quadrotor efficiency through adaptive control is essential for addressing the crit- ical need for Fault-Tolerant Control (FTC) in the presence of operational uncertainties and component inefficiencies. Conventional adaptive FTC strategies typically assume that uncer- tainties are bounded by a constant known a priori. However, this assumption is inadequate when dealing with imprecise inertial system parameters, leading to state-dependent uncertain- ties that deviate from this model. When left unaddressed, such uncertainties can result in insta- bility, particularly under actuator faults, posing significant risks to the reliability of quadrotor systems. To overcome these challenges, the proposed adaptive FTC method effectively mitigates ac- tuator faults and addresses unknown state-dependent uncertainties through the implementation of carefully designed adaptive laws. This approach incorporates real-time fault detection and dynamic control allocation, which together prevent overly conservative control applications, ensuring that the quadrotor maintains optimal performance even under adverse conditions. By dynamically adjusting control efforts based on real-time data, the system is capable of respond- ing robustly to varying fault scenarios, thereby enhancing overall operational resilience. The stability of the closed-loop system is rigorously analyzed using Lyapunov theory, which provides a strong theoretical foundation for the proposed framework. Analytical studies con- firm that the adaptive FTC maintains stability across a wide range of operational scenarios, en- suring the quadrotor’s reliability. The effectiveness of this solution is further validated through comprehensive simulations on a realistic quadrotor model, demonstrating significant improve- ments over existing methods in terms of both stability and efficiency. This research highlights the importance of adaptive control strategies in maintaining reliable and safe quadrotor operations, setting the stage for future advancements in UAV fault tolerance and control systems. By addressing the complexities of state-dependent uncertainties and actu- ator faults, the study contributes to the broader field of autonomous aerial vehicles, ultimately enhancing the safety and effectiveness of UAV deployments in various applications.

Abstract

Music has therapeutic potential for individuals with Autism Spectrum Disorder (ASD). Traditional music therapy often lacks personalization, with music selected by therapists rather than tailored to individual preferences. Current studies on ASD individuals’ music preferences often overlook cultural inclusivity and the impact of lyrics. Moreover, existing recommendation systems do not consider music preferences tailored for therapeutic goals. This thesis explores how music aids emotional regulation, cognitive functioning, and social interaction in individuals with ASD, emphasizing cultural inclusivity. The study includes analyses of music specific community for neurotypical individuals (NT) and ASD communities on Reddit, cross-cultural comparisons between Non-Indian and Indian populations, a pilot study with Indian children with ASD, and the development of a personalized Music Recommendation System. Reddit analysis revealed diverse music preferences, underscoring the need for personalized interventions. Cross-cultural comparisons showed non-Indian participants preferred Rock and Western Classical music, while Indian participants favored Indian film and Devotional music, reflecting cultural influences on emotional expression. The pilot study with Indian children identified diverse music preferences, including Indian Devotional music, Nursery rhymes, and Indian film music. Listening to preferred music indicated positive behavioral changes, such as reduced anxiety and improved social interaction, highlighting music’s therapeutic potential. The GOAL (Genre, Optimization, Acoustic Features) based Music Recommendation System offers personalized suggestions based on genres, acoustic features, and lyrical themes relevant to emotional regulation, cognitive functioning, and social skills. By categorizing goals into emotional, cognitive, and social functions, the system aligns with diverse ASD needs and provides personalized music recommendations using similarity metrics. Evaluation metrics indicated the model accurately recommends relevant music, achieving a high proportion of liked music (precision) and capturing a significant portion of preferred music (recall). Behavioral changes showed effectiveness in mood management, focus, sensory load regulation, and sleep regularity, with mixed results in communication and social skills. In conclusion, this thesis underscores the importance of recognizing and accommodating ASD individuals’ music preferences in therapy. Despite limitations such as small sample sizes and subjective reports, findings offer insights into music’s therapeutic potential for ASD. Future research should include larger, diverse samples and longitudinal studies to enhance the proposed system’s effectiveness.

Abstract

Scene text conveys important information to drivers and pedestrians, serving as an indispensable means of communication in various environments. Scene text contains information regarding the speed limit, route information, rest stops, and exits, among other important information. It is important for drivers and passengers to understand this information for a safe and efficient journey. However, outdoor scenes are cluttered with text, distracting drivers and making it hard to focus on what matters, potentially compromising their ability to focus on essential details and navigate safely. Recognising scene text in motion aggravates this challenge, as textual cues transiently appear and necessitate early detection at a distance. Driving scenarios introduce additional complexities, including occlusions, motion blur, perspective distortions, and varying text sizes, further complicating scene text understanding. In this thesis, we look at improving scene text understanding in diving scenarios through video question answering and analysing the present state of scene text detection, recognition and tracking: (i) We introduce a new video questions answering task and dataset that requires an understanding of text and road signs in driving videos to answer the questions. (ii) We look at the current state of scene text detection, tracking and recognition in the driving domain through the RoadText-1K competition. (iii) We explore detection and recognition in special cases of occlusions, a common yet under-explored complication in real-world driving scenarios. By focusing on these areas, the thesis contributes to advancing scene text analysis methodologies, offering insights and solutions that are imperative for developing more intelligent and responsive driver assistance systems.

Abstract

This thesis addresses the critical challenge of monitoring water flow to support efficient water resource management and conservation. It presents an innovative approach to developing a sustainable water management system through an Internet of Things (IoT)-based solution. The primary objective is to retrofit existing analog water meters using affordable, off-the-shelf electronic components, transforming them into smart devices capable of real-time data collection and analysis. A distinctive feature of this system is its use of edge computation, which allows meter readings to be extracted directly from images captured by an onboard camera. This local processing approach minimizes latency and reduces the need for continuous cloud communication, enhancing system responsiveness and reducing data transmission costs. Energy efficiency and sustainability are critical components of this design. To support autonomous operation, the system is powered by solar-charged batteries, with strategies implemented to preserve charge-discharge cycles even in challenging environmental conditions. Additionally, an automated broadcasting technique facilitates firmware updates across multiple devices simultaneously, reducing the need for manual intervention and enhancing overall maintenance efficiency. The effectiveness and reliability of the system were validated through an extensive field deployment involving 15 smart water meter nodes installed across various locations at IIIT-H campus and monitored continuously for over 2.5 years. This long-term deployment provided valuable real-world data, allowing for detailed characterization of water usage patterns across different environments. The stable operation of these nodes over a prolonged period demonstrated the robustness and durability of the design, supporting its potential for broader application in diverse settings. An essential feature of the system is its energy-harvesting capability, which minimizes reliance on external power sources and significantly reduces maintenance requirements. This feature aligns with sustainable energy practices and promotes an eco-friendly approach to water resource management. By integrating efficient data acquisition, local processing, and cloud-based storage and analysis, this IoTbased solution offers a comprehensive, scalable, and sustainable approach to advancing water resource management and supporting global conservation efforts.