Abstract

This thesis focuses on advancing automated video editing by analyzing raw, unedited footage to extract essential information such as speaker detection, video saliency, and dialogue interpretation. At the core of this work is EditIQ, an automated video editing pipeline that leverages speaker cues, saliency predictions, and large language model (LLM)-based dialogue understanding to optimize shot selection—the critical step in the editing process. The study begins with a comprehensive assessment of active speaker detection techniques tailored for automated editing. Using the BBC Old School Dataset, annotated with active speaker information, we propose a robust audio-based nearest-neighbor algorithm that integrates facial and audio features. This approach reliably identifies speakers even under challenging conditions such as occlusions and noise, outperforming existing methods and closely aligning with manual annotations. In the domain of video saliency prediction, we present ViNet-S and ViNet-A, compact yet effective models designed to predict saliency maps and identify salient regions in video frames. These models are computationally efficient, balancing high accuracy with reduced model complexity. Starting with a static, wide-angle camera feed, EditIQ generates multiple virtual camera feeds, mimicking a team of cinematographers. Speaker detection, saliency-based scene understanding, and LLMsdriven dialogue analysis guide shot selection, which is formulated as an energy minimization problem. This optimization ensures cinematic coherence, smooth transitions, and narrative clarity in the final output. The efficacy of EditIQ is validated through a psychophysical study involving twenty participants using the BBC Old School dataset. Results demonstrate EditIQ’s ability to produce aesthetically compelling and narratively coherent edits, surpassing competing baselines and showcasing its potential to transform raw footage into polished cinematic narratives.

Abstract

Language models often exhibit undesirable behavior, e.g., generating toxic or gender-biased text. In the case of neural language models, an encoding of the undesirable behavior is often present in the model’s representations. Thus, one natural (and common) approach to prevent the model from exhibiting undesirable behavior is to steer the model’s representations in a manner that reduces the probability of it generating undesirable text. In this thesis, we present work that investigates the formal and empirical properties of steering functions, i.e., transformation of the neural language model’s representations that alter its behavior. First, we derive two optimal, in the least-squares sense, affine steering functions under different constraints. Our theory provides justification for existing approaches and offers a novel, improved steering approach. Second, we offer a series of experiments that demonstrate the empirical effectiveness of the methods in mitigating bias and reducing toxic generation.

Abstract

n recent years, computer vision has made remarkable progress in understanding visual scenes, in- cluding tasks such as object detection, human pose estimation, semantic segmentation, and instance segmentation. These advancements are largely driven by high-capacity models, such as deep neural networks, trained in fully supervised settings with large-scale labeled data sets. However, reliance on extensive annotations poses scalability challenges due to the significant human effort required to create these data sets. Fine-grained annotations, such as pixel-level segmentation masks, keypoint coordinates for pose estimation, or detailed object instance boundaries, provide the high precision needed for many tasks but are extremely time-consuming and costly to produce. Coarse annotations, on the other hand, such as image-level labels or approximate scribbles, are much easier and faster to create but lack the granularity required for detailed model supervision. To address these challenges, researchers have increasingly explored alternatives to traditional su- pervised learning, with weakly supervised learning emerging as a promising approach. This approach mitigates annotation costs by utilizing coarse annotations (cheaper and less detailed) during training rather than the fine-grained annotations required at the output stage during testing. Despite its poten- tial, weakly supervised learning faces challenges in transferring information from coarse annotations to fine-grained predictions, often encountering ambiguity and uncertainty during this process. Existing methods rely on various priors and heuristics to refine annotations, which are then used to train models for specific tasks. This involves managing uncertainty in latent variables during training and ensuring accurate predictions for both latent and output variables at test time. This thesis introduces a unified approach to weakly supervised learning in computer vision, address- ing tasks such as human pose estimation, object detection, and instance segmentation. Central to this work is a framework based on the dissimilarity coefficient loss, which models uncertainty in the loca- tion of objects and human poses using coarse annotations. The approach employs two key probability distributions: • Conditional Distribution: Captures output probabilities using coarse annotations (e.g., action la- bels, image-level labels, object counts), modeled with deep generative models for efficient sam- pling. • Prediction Distribution: Provides test-time predictions independent of coarse annotations.The framework minimizes the difference between these distributions using the dissimilarity coeffi- cient loss, facilitating the transfer of information from coarse annotations to accurate predictions. This methodology is consistently applied across diverse computer vision tasks, showcasing its versatility. The efficacy of the proposed framework is demonstrated across three progressively complex visual scene recognition tasks: • Human Pose Estimation: A probabilistic framework is introduced for learning human poses from still images using data sets with costly ground-truth pose annotations and inexpensive action la- bels. By aligning the conditional and prediction distributions through the dissimilarity coefficient loss, the method achieves significant improvements over baselines on the MPII and JHMDB data sets, effectively leveraging action information. • Object Detection: The framework addresses weakly supervised object detection (WSOD) by mod- eling uncertainty in object locations using a dissimilarity coefficient-based objective. Leveraging discrete generative models, it efficiently samples from annotation-aware conditional distributions and integrates coarse annotations, such as image-level labels, object counts, points, and scribbles. Spatial cluster regularization and curriculum learning further enhance performance, achieving state-of-the-art results on benchmarks like PASCAL VOC and MS COCO. • Instance Segmentation: The framework models uncertainty in pseudo-label generation using se- mantic class-aware, boundary-aware, and annotation-consistent higher-order terms. By aligning conditional and prediction distributions, it generates accurate pseudo-labels and trains Mask R- CNN-like architectures effectively. Experiments on the PASCAL VOC 2012 data set demonstrate state-of-the-art performance, with improved object boundary alignment and significant gains over baselines

Abstract

The increasing frequency and severity of crowd-related disasters in complex public spaces underscore a critical need for robust, predictive, and adaptable evacuation planning systems. This thesis details the design, implementation, and evaluation of the Strategic Evacuation System (SES), a comprehensive simulation framework engineered to model and analyze human strategic behaviour during emergency evacuations. The core of the SES is a Game-Theoretic agent-based model (GT-ABM) that simulates the decision-making of heterogeneous evacuees, capable of operating under both Non-Cooperative (selfinterested) and cooperative paradigms. To ground these interactions in a realistic spatial context, this work introduces a semantic extension to the IndoorGML standard, enabling the encoding of dynamic, evacuation-specific properties such as path risk, congestion, and accessibility rights directly within the spatial data model. The system architecture embeds a sophisticated decision-making engine that integrates key GameTheoretic concepts to achieve behavioural realism. These include Nash Equilibria to analyze selfish outcomes, Bayesian Games for decision-making under incomplete information, Signalling Games to model the impact of external guidance from authorities, and Evolutionarily Stable Strategies (ESS) to evaluate the stability and efficiency of evacuation protocols for diverse populations. The efficacy of this system and the validity of its underlying models are rigorously evaluated through a series of case studies simulating high-density bottleneck congestion in a hostel, complex multi-path failure modes in a multiplex, and the evacuation dynamics of heterogeneous populations with varying fitness levels. The results provide clear, quantitative evidence demonstrating the catastrophic failure modes of purely selfish behaviour - a real-world manifestation of the "Price of Anarchy" where uncoordinated decisions lead to systemic collapse and high numbers of trapped individuals. Conversely, the simulations prove that cooperative strategies, aided by informative Signalling, achieve vastly superior outcomes, including 100% evacuation success and significantly reduced clearance times. The analysis of Evolutionarily Stable Strategies further reveals non-intuitive pathways to optimizing evacuation protocols beyond mere survival, enhancing overall efficiency. This research establishes a validated, extensible platform for both academic Modelling and practical safety engineering. It concludes by outlining a forward-looking vision for the systems deployment as the core of a real-world, large-scale crowd management application for the Tirumala Temple in India, a site managing over 25.5 million visitors annually. By integrating the SES with AI-driven monitoring and IoT data streams a paradigm justified by the successful AI implementation at the Tirumala Temple this work lays the potential foundation for the next generation of intelligent socio-technical systems for urban and public safety.

Abstract

Intermittent Water Supply (IWS) systems, prevalent in many developing regions, pose significant challenges for urban water management due to their scheduled delivery and the resulting need for consumers to store water for non-supply periods. The lack of high-frequency, building-level consumption data in such systems hinders accurate demand profiling, leakage detection, and informed infrastructure planning. This thesis addresses these gaps by developing and deploying scalable, Internet of Things (IoT) enabled strategies to capture and analyse water consumption patterns in IWS settings, with a particular focus on the Indian context. The work is structured in two parts: first, improving the accuracy and reliability of IoT-based smart retrofit devices for analog water meter digitization; and second, constructing and analysing water consumption curves at different temporal resolutions for diverse building types using the collected data. Retrofitting existing analog water meters is a cost-effective approach to enable high-frequency, automated water usage monitoring without replacing established infrastructure. While image-based meter reading using IoT devices has shown promise, real-world deployments encounter significant challenges such as dew accumulation, scratches, smudges, and insect intrusion, all of which can degrade image quality and lead to digit detection errors. To address these issues, this thesis introduces a lightweight Hamming distance-based refinement algorithm that systematically corrects digit misclassifications by leveraging the expected range of meter readings and minimizing the number of differing digits. The algorithm is computationally efficient, requiring only basic arithmetic operations, and is thus well-suited for real-time deployment on edge devices like the Raspberry Pi. To further enhance data quality, a webbased annotation tool was developed, enabling rapid, targeted manual validation of flagged anomalies and supporting the efficient correction and curation of large datasets. Building on these advancements, the thesis proposes two innovative, non-intrusive IoT-based strategies for capturing water consumption in IWS-serviced buildings. The first strategy involves installing smart flow meters at the outlets of Over Head Tank (OHT) to directly measure consumption. The second combines flow meters at the inlet with water level sensors inside OHTs, using mass conservation principles to infer consumption at high temporal resolution. These approaches minimize the need for intrusive instrumentation and are readily scalable to a variety of building types commonly found in Indian urban settings. A comprehensive field study was conducted at International Institute of Information TechnologyHyderabad (IIIT-H), where seven IoT nodes were deployed across five buildings, including hostels and classroom blocks. Over 125,000 data points were collected at three-minute intervals over a period of two months, capturing detailed water usage dynamics. The data underwent rigorous cleaning, refinement, and synchronization, leveraging the developed algorithms and annotation tools to ensure high integrity and accuracy. Using a bottom-up approach, the thesis constructs instantaneous, diurnal, and weekly water consumption curves for each building, normalized using peak factors to facilitate comparison of demand volatility across user categories. The resulting consumption patterns reveal distinct signatures for residential and non-residential buildings, highlight the impact of scheduled events (such as sports screenings and hackathons), and quantify temporal variations in demand. These high-resolution insights enable utilities and building managers to detect leaks and unauthorized usage, optimize supply schedules, and inform the design of future-ready water distribution networks. The methodologies and findings also provide a robust foundation for transitioning from IWS to CWS systems and for upgrading conventional buildings to smart, resource-efficient infrastructure.

Abstract

Malaria, caused by Plasmodium falciparum, remains a major global health challenge, requiring both a deeper understanding of parasite biology and innovative therapeutic strategies. This thesis presents a dual-pronged computational approach to address this challenge by combining transcriptomic analysis of Plasmodium falciparum with network-driven drug repurposing methodologies to address these gaps. First, we analyze single-cell RNA sequencing (scRNA-seq) data from the Malarial Cell Atlas to gain comprehensive insights into P. falciparum’s complex life cycle. Using feature selection techniques and a neural network classifier, we extract high-confidence gene sets that capture stage-specific signatures with high accuracy and outperform the gene set, confirming the capture of distinctive stage-specific signatures. Functional enrichment and pathway analyses further validate their biological relevance, linking distinct gene sets to immune evasion, haemoglobin digestion, merozoite invasion, and sexual differentiation. Next, we implemented a network-driven drug repurposing framework to identify repurposable drugs for malaria. We constructed an integrated graph combining Protein-Protein Interaction networks, Drug-Target Protein graphs, Disease-Protein associations, and Drug-Disease links to predict potential antimalarial drug candidates. Comparative analysis of several Graph Neural Network architectures, including Graph Convolutional Networks, Graph Attention Networks, GraphSAGE, and Graph Isomorphism Networks, showed comparable performance with AUC-ROC values around 0.98. However, when the task was framed as a recommendation problem using Matrix Factorization with side information, performance decreased significantly, highlighting the critical importance of incorporating protein-protein interactions in the modelling process. These findings underscore the power of combining transcriptomic insights with computational drug discovery. By refining gene selection for parasite stage characterization and leveraging network-based drug repurposing, this work provides a data-driven framework for identifying potential malaria therapeutics. Future research should focus on experimental validation and extending these methodologies to other infectious diseases.

Abstract

Human intelligence relies on compositional generalization: the ability to interpret novel situations by flexibly combining familiar concepts and relational structures. This thesis investigates compositionality in vision-language models (VLMs), focusing on their ability to understand and generalise across visual (images, videos) and linguistic inputs. In the first part, we introduce VELOCITI, a benchmark for evaluating compositional understanding in video-language models through a suite of entailment tasks. Unlike prior compositionality benchmarks constrained to single-agent videos, VELOCITI captures the complexity of real-world videos involving multiple agents and dynamic interactions. VELOCITI assesses how well models recognize and bind agents, actions, and temporal events using both text-inspired and in-video counterfactual negations. In the second part, we probe the internal activations of VLMs to understand how concepts in an image are bound to their attributes and references in text. Extending the Binding ID mechanism in language models, we demonstrate that VLMs construct binding ID vectors in the activations of both image tokens and their textual references, enabling in-context concept association. Together, these contributions advance our understanding of compositional reasoning in VLMs and offer tools for probing their capabilities.

Abstract

This thesis examines the utilisation of hybrid methodologies, integrating traditional textual analysis with computational techniques, to optimise research efficiency in the analysis of political discourse. This endeavour aligns with the ongoing paradigm shift within the Digital Humanities, driven by advancements in computational and machine learning technologies. Specifically, two distinct projects analyse the public addresses of the incumbent Prime Minister, Narendra Modi. The first project investigates Mann ki Baat, a series of radio broadcasts initiated during the first Modi administration. This study aims to ascertain the primary objective of the series. The second project analyses the Prime Minister’s Independence Day speeches (15th August), exploring the contrasts in narrative style and content between the two terms of the current administration and the rationale behind these perceived changes. The first study examining the Mann ki Baat radio broadcasts utilises a suite of Natural Language Processing (NLP) techniques to generate initial observations. Subsequently, classical textual analysis is applied to a carefully curated corpus of 25 speeches, selected for their temporal alignment with significant socio-political events in India, to validate, substantiate, and extend these findings. The research demonstrates that the series functions as an aesthetic imperative, representing a large-scale governmental initiative that uses state machinery and financial resources to propagate the values and ideals of the BJP as a political entity over the overarching NDA government. The second study divides Narendra Modi’s ten Indian Independence Day addresses according to his terms of office. To facilitate rigorous analysis, a comprehensive annotation procedure was implemented, wherein the thematic content of each sentence was systematically categorised. Subsequently, a comparative analysis was performed, contrasting the content and narrative styles of the two sets of speeches. The findings reveal a significant thematic transition: from an emphasis on inclusivity during the first term to a focus on infrastructural development and national self-sufficiency in the second. Furthermore, discernible shifts in narrative style were observed, including a consistent augmentation of monoreligious allusions and direct appeals to a gendered audience. By integrating the conclusions derived from both projects, albeit obtained through different methodologies, a coherent understanding of the overarching narrative that the government hasarticulated during its tenure in power can be established. Furthermore, this research demonstrates the feasibility of employing hybrid methodologies in the analysis of political rhetoric.

Abstract

This thesis addresses the problem of achieving fairness in probabilistic binary classification in the presence of binary protected groups. In many practical scenarios, classifiers assign scores, and decision thresholds are applied post hoc based on the desired trade-off between false positives and false negatives. However, the use of a fixed classifier with arbitrary thresholds may result in disparate treatment across protected groups. To ensure equitable outcomes, we introduce a fairness criterion, denoted εp-Equalized ROC, which requires that the Lp norm between the false positive rates (FPRs) and true positive rates (TPRs) of the protected groups remains within a predefined bound ε—regardless of the threshold chosen. To operationalize this notion of fairness, we propose a post-processing approach that transforms the outputs of an existing (potentially unfair) classifier into a randomized and threshold-agnostic classifier that satisfies ε1-Equalized ROC. Central to this approach is a novel threshold query model over ROC curves, which enables controlled manipulation of the classifier’s behavior across groups. We characterize the inherent trade-off between fairness and utility by deriving a theoretical lower bound on AUC loss and proving that some AUC reduction is inevitable under fairness constraints. To realize our fairness objective in practice, we develop a linear-time algorithm, FROC, which guarantees ε1-Equalized ROCcompliance. We prove that under reasonable assumptions, FROC achieves optimality with respect to the minimal necessary AUC degradation. Extensive experiments on real-world datasets—using classifiers such as Weighted Ensemble L2, Random Forest (Gini), and FNNC—demonstrate that FROC is both theoretically sound and empirically effective across multiple fairness and performance metrics. The thesis is structured into six chapters, beginning with an introduction to algorithmic bias and culminating in theoretical analysis, algorithmic development, and empirical validation. Through this work, we contribute a principled and practical framework for ensuring fairness in score-based classification systems

Abstract

The need for photorealistic head avatars has risen in the past decades, owing to the rising interest in the AR/VR media formats. An accurate representation of the head will be required in the near future, which is essential to facilitate communication between users, essentially enabling telepresence. The need for an improved in-person form of remote communication was made more clear during the recent COVID-19 pandemic and the ensuing lockdown, where millions of people had to stay away from their families and workplace for an extensive period of time. Besides, realistic facial avatars have proved immensely helpful in the movie and gaming industry, where they have often been used to either modify the actors’ appearance itself, or to drive an entirely virtual but realistically looking digital character, depending on the demands of the narrative. Capturing and reconstructing a realistic-looking head-avatar is not trivial, and requires an expensive setup of multiple synced cameras and lights, and a mathematical understanding of how light interacts with the skin, hair, cornea, etc. The capture of each subject is laborious and time-consuming. The creation of digital faces that are indistinguishable from real ones is a formidable challenge due to the ”uncanny valley” phenomenon, where even minor deviations from realistic appearance can render a digital face unsettling to human observers. However, to achieve the applications of realistic head avatars in telepresence and AR/VR, the capture and thus creation of realistic digital replicas must be made accessible. Therefore, a need has arisen to search for methods that can reconstruct and create digital replicas that are photorealistic but also cheap. Our thesis aims to tackle this problem statement in two ways, one from the perspective of digital replica generation and the other from the perspective of creating a digital replica through reconstruction. Our initial approach to make the creation of digital replicas efficient is a textured head generation method conditioned on a descriptive text. We aim to create a method that can generate a realistic-looking head avatar from a text description in an efficient manner, without requiring the manual intervention of artists or the use of highly specialised software like Blender or Maya. Therefore, it can generate textured head assets within seconds. However, the texture-based synthesis approach suffered from reduced realism because of the effects of baked-in lighting. Therefore, an approach is required that could construct a head avatar along with accurate material properties, such that it can be placed in any environment. Our following work proposes a method to reconstruct a relightable head avatar using just a smartphone. Existing works that focus on creating a digital replica from a smartphone do not create a relightable head avatar or require a large amount of data prior to generating a head avatar with appropriate material properties. Our proposed method introduces a novel capture strategy that can act as a smallscale replacement of lightstage-based capture, while only utilising a smartphone. We also propose an efficient head representation which allows real-time rendering and relighting, making it suitable for the application of telepresence. We evaluate our proposed methods on public datasets, and conclude that our methods outperform state-of-the-art methods, both in the realms of text-based generation and capture-based reconstruction. We also release our own dataset, recorded using our novel capture setup. Additionally, we discuss our methods’ limitations, offering insights into potential improvements and outlining promising future research directions. We hope that this thesis will substantially impact the field and accelerate progress in 3D Head generation and reconstruction.