Abstract

A Completely Automated Public Turing Test to Tell Computers and Humans Apart (CAPTCHA) is one of the primary barriers between notorious bots and legitimate human users. However, advancements in Artificial Intelligence (AI) have enabled malicious bots to circumvent CAPTCHA challenges effectively. As a result, several types of CAPTCHA have been rendered ineffective. In this work, we introduce Swiss Cheese CAPTCHA, a novel sensor-based solution designed to be easily solvable by humans while presenting multiple obstructions for bots (similar to the Swiss Cheese Model) even when the sensor outputs can be predicted and interfered with. We leverage a range of human cognitive abilities and Generic Sensor API in modern devices to provide robust protection against automated attacks by making it more computationally expensive for bots to produce a valid answer within a stipulated time. We conducted two user studies to assess our proposal’s effectiveness: one involving 116 participants to assess the likability and improvise the design, and the other, with 107 participants, to investigate the impact of improvised design changes on cognitive abilities. Our results from these studies show an average completion time of 4.76 seconds and 6.12 seconds, with a success rate of 90.3% and 83.25%, respectively. By analyzing the 2141 resultant trajectories from both user studies, we assess the learnability, error recovery rate, efficiency, and satisfaction of users using the scheme. Finally, we devise an automated attack against our proposal to analyze its security in the real world; we find the probability of attack success is low. We also make our dataset available for further research.

Abstract

This thesis presents a comprehensive exploration of Multi-Object Multi-Part Scene Parsing in 2D images, showcasing significant advancements through two novel approaches, FLOAT and OLAF, each tailored to enhance scene parsing performance and scalability. The first paper introduces FLOAT, a factorized label space framework designed to independently predict object categories and part attributes, thereby simplifying the segmentation task and enhancing scalability. Notably, FLOAT incorporates a unique ’zoom’ refinement technique at inference time, significantly elevating segmentation accuracy, particularly for smaller objects and parts. Empirical results on the Pascal-Part datasets underscore FLOAT’s superior performance, achieving notable improvements in mean Intersection Over Union (mIOU) and segmentation quality IOU (sqIOU), especially on the most comprehensive PascalPart-201 dataset, reflecting its effectiveness in handling diverse and complex scenes. The second paper delves into OLAF, a plug-and-play methodology that augments traditional RGB inputs with object-based structural cues to better capture the complexities of scene structures. This approach leverages a weight adaptation technique, allowing pre-trained RGB models to seamlessly integrate augmented data, thus stabilizing the optimization process. Additionally, the introduction of the LDF encoder module aids in providing low-level dense feature guidance, enhancing the segmentation of smaller parts. OLAF demonstrates its versatility across various architectures and datasets, achieving significant mIOU gains on multiple Pascal-Part benchmarks, highlighting its broad applicability and robust performance enhancements in challenging segmentation scenarios.

Abstract

In Indian urban and rural driving scenarios, small objects are pervasive and often crucial for safe navigation. These objects can include pedestrians crossing roads, children playing near streets, cyclists, stray animals, as well as small vehicles like scooters and motorbikes. Additionally, traffic signs, signal lights, potholes, and road markings (such as lane dividers or zebra crossings) are often small in size but essential for driving decisions. In such contexts, missing or inaccurately segmenting these small objects can lead to critical errors in detection, causing accidents or delays in the vehicle’s decision-making process. Automated understanding of such objects need detection and segmentation to start with. Semantic Segmentation is a critical task in computer vision with a wide range of applications. The objective is to partition an image—a collection of pixels—into distinct labeled regions, each corresponding to specific objects or parts of the scene. This process is crucial for scene understanding and enables the localization of objects within the image. Over time, significant progress has been made in semantic segmentation, especially with the advent of deep learning. The advances in this area have revolutionized computer vision, pushing beyond traditional methods and achieving remarkable improvements in performance. When discussing semantic segmentation, we often focus on datasets, the objects within those datasets, and their corresponding segmentations. While many datasets exist for road scenarios, particularly those representing Western road conditions, there is relatively little research on road conditions specific to India. One notable exception is the Indian Driving Dataset (IDD), a dataset specifically designed for semantic segmentation of Indian road scenarios. Road and driving datasets typically contain objects of varying sizes within each class label. These objects can be broadly categorized into three types: small, medium, and large. The importance of segmentation is well understood across several domains such as medical imaging, autonomous vehicles, aerial imagery, robotics, surveillance, and industrial automation. However, one of the most challenging problems in segmentation is the segmentation of small objects. Small object segmentation is particularly difficult due to factors such as (i) the limited number of pixels representing small objects, (ii) class imbalance during training, and (iii) the inherent challenges posed by small object representations. These factors hinder the performance of deep learning architectures, making it harder for modern techniques to accurately handle small objects. This issue becomes evident when evaluating deep learning models, as performance metrics tend to be significantly worse for small objects. In this study, we delve into the challenges of segmenting small objects and explore potential strategies for improving algorithm design for this task. It is widely acknowledged that loss functions play a key role in segmentation performance, alongside the architecture of deep learning models. Indeed, the segmentation of small objects is highly influenced by the choice of loss function during model training. Various small objects commonly encountered in road scenarios are highlighted in this study, emphasizing their importance in Indian road environments. Objects such as pedestrians, traffic signs, and traffic lights—despite their small size—are crucial for tasks like autonomous navigation and driver assistance systems. The Indian Driving Dataset (IDD) offers a valuable resource, containing a wide range of small objects across different class labels, some of which dominate specific categories. These objects are captured and annotated in real-world, unstructured environments, making this dataset particularly relevant for segmentation tasks in Indian road scenarios. Therefore, our focus in this study is primarily on detection and segmentation of small objects within the IDD dataset and explore the impact of loss functions.

Abstract

Principal component analysis(PCA) is one of the most popular linear dimensionality reduction techniques in machine learning. In this thesis, we present a method for performing PCA on encrypted data using a homomorphic encryption scheme. We used the CKKS homomorphic encryption scheme for our purpose since it is most suitable for machine learning tasks allowing approximate computations on real numbers. In this thesis, we propose a new Homomorphic PCA(HPCA) algorithm that performs PCA. Unlike previously proposed algorithms, our HPCA algorithm is non-interactive in nature. This requires an approximation of the inverse sqrt function. This thesis presents two methods to approximate and securely perform the inverse sqrt function using CKKS homomorphic encryption scheme - Constrained Linear Regression and Pivot-Tangent method. Since the CKKS homomorphic scheme allows only the computation of polynomial functions, we propose a method to approximate the inverse sqrt function polynomially. Ultimately, we implement our approach for the inverse sqrt function. We provide an implementation of our method for the inverse sqrt function. We use the inverse sqrt approximations in our HPCA algorithm to make it non-interactive. In the end, we present the experimental results of our proposed HPCA algorithm on a few datasets computing a few principal components. We measure the R2 score on the reconstructed data and use it as an evaluation metric for our HPCA algorithm.

Abstract

This thesis explores the dual objectives of reviewing material characteristics of soft polymers for microfluidic and Lab-on-a-Chip (LOC) devices and developing a cost-effective, flexible microheater for LOC applications. Soft materials such as polydimethylsiloxane (PDMS), hydrogels, thermoplas- tic elastomers, and conducting polymers are analyzed for their mechanical, thermal, chemical, optical, and biocompatibility properties. These polymers play a crucial role in LOC systems due to their abil- ity to provide flexibility, durability, and compatibility with biological samples. The review delves into their fabrication techniques, solubility in common solvents, adhesion, and surface modification meth- ods, highlighting their relevance for applications such as diagnostics, cell culture, and chemical assays. Experimental studies further assess their optical properties and cytotoxicity, offering a comprehensive understanding of their potential for use in biomedical and chemical systems. By comparing the advan- tages and limitations of these materials, this work aids in optimizing material selection for enhanced performance and sustainability in LOC applications. The developed flexible microheater utilizes nichrome wire embedded in a PDMS substrate, chosen for its affordability and durability. The microheater operates within a temperature range of 40°C to 150°C, with precise control achieved through a pulse-width modulation (PWM)-based circuit and feed- back from a 100k Ω NTC thermistor. This setup provides closed-loop temperature regulation, ensuring stability and adaptability for diverse applications. The microheater’s efficacy is demonstrated in two sig- nificant applications: glucose detection and the synthesis of silver nanoparticles (AgNPs). In the glucose detection application, the microheater integrated into a microwell device reliably facilitates Benedict’s test, confirming its potential for point-of-care diagnostics. The microheater is also employed in the syn- thesis of AgNPs using green chemistry principles, utilizing ascorbic acid as a reducing agent and sodium citrate as a stabilizer. This process yields spherical nanoparticles with UV-Vis absorption peaks charac- teristic of AgNPs (between 405 nm and 411 nm). Both applications underline the microheater’s capacity to deliver consistent performance while maintaining low fabrication costs. Experimental results validate the heater’s capability to reach steady-state temperatures quickly and efficiently. The integration of the PWM-based control system ensures rapid response times and enhanced accuracy, addressing challenges in conventional heating systems. Furthermore, the cost of the microheater device is kept below INR 1000, making it an accessible solution for resource-constrained settings.

Abstract

In the field of artificial intelligence, the generation of human-like motion from natural language descriptions has garnered increasing attention across various research domains. Computer vision focuses on understanding and replicating visual cues for motion, while computer graphics aims to create and edit visually realistic animations. Similarly, multimedia research explores the intersection of data modalities, such as text, motion, and image, to enhance user experiences. Robotics and human-computer interaction are pivotal areas where language-driven motion systems improve the autonomy and responsiveness of machines, facilitating more efficient and meaningful human-robot interactions. Despite its significance, existing approaches still encounter significant difficulties, particularly when generating motions from unseen or novel text descriptions. These models often lack the ability to fully capture intricate, low-level motion nuances that go beyond basic action labels. This limitation arises from the reliance on brief and simplistic textual descriptions, which fail to convey the complex and fine-grained characteristics of human motion, resulting in less diverse and realistic outputs. As a result, the generated motions frequently lack the subtlety and depth required for more dynamic and context-specific applications. This thesis introduces two key contributions to overcome these limitations and advance text-conditioned human motion generation. First, we present Action-GPT, a novel framework aimed at significantly enhancing text-based action generation models by incorporating Large Language Models (LLMs). Traditional motion capture datasets tend to provide action descriptions that are brief and minimalistic, often failing to convey the full range of complexities involved in human movement. Such sparse descriptions limit the ability of models to generate diverse and nuanced motion sequences. Action-GPT leverages LLMs to create richer, more detailed descriptions of actions, capturing finer aspects of movement. By doing so, it improves the alignment between text and motion spaces, enabling models to generate more precise and contextually accurate motion sequences. This framework is designed to work with both stochastic models (e.g., VAE-based) and deterministic models offering flexibility across different types of motion generation architectures. Experimental results demonstrate that Action-GPT not only enhances the quality of synthesized motions—both in terms of realism and diversity—but also excels in zero-shot generation, effectively handling previously unseen text descriptions. Second, we introduce MoRAG, a sophisticated retrieval-augmented generation strategy tailored to enhance the performance of motion diffusion models. MoRAG adopts a multi-part fusion retrieval mechanism that allows for improved generalization of motion retrieval across a wide range of language inputs, addressing the limitations of current retrieval methods that struggle with unseen or atypical descriptions. By incorporating low-level, part-specific motion details into the retrieval process, MoRAG constructs more accurate and varied motion sequences. The retrieval process is refined by prompting LLMs to handle issues like spelling errors, rephrasing, and ambiguous language, ensuring that the retrieved motions are contextually relevant and diverse. These augmented motion samples are then used as additional knowledge within the motion generation pipeline, enhancing the system’s ability to generate complex and realistic motions from diverse textual inputs. This retrieval-augmented approach increases both the robustness and generalization capacity of the motion generation models, making them more adaptable to complex and unseen scenarios. Together, these contributions represent a substantial advancement in text-conditioned human motion generation. By refining both the action description process and the motion retrieval strategy, this work enhances the ability of models to generate diverse, realistic motions from natural language inputs, particularly in zero-shot settings and when handling detailed, complex descriptions.

Abstract

The performance of wireless communication systems is fundamentally constrained by the propagation environment, which is uncertain, volatile, and constantly changing due to factors such as multipath fading, doppler shift, path loss, etc. Legacy solutions such as multiple input multiple output, relays, etc., have circumvented these issues to a certain extent in order to establish reliable communication links. Nonetheless, as communication systems are heading towards a future with data rate requirements in the order of gigabits/second and latencies in the order of microseconds, efficient system design is the need of the hour. Especially, solutions that merely circumvent the ill-effects of wireless propagation environments are no longer sufficient. Towards this, a new technology named Reconfigurable Intelligent Surface (RIS) is emerging as a potential solution that has the capability to alter the propagation environment to a large extent, and hence is envisioned to be a part of next-generation cellular standards. There are many research directions for the implementation of RIS-aided communication systems, such as RIS hardware design, channel modeling, channel estimation, optimal beamforming, capacity and outage analysis, optimal RIS deployment, etc. These directions must be explored in-depth in order to realize the benefits of such RIS-aided communication systems fully. This thesis aims to explore RIS-aided communication systems design and their performance analysis in terms of capacity and outage, with a special emphasis on proposing low-complexity beamforming solutions and their performance analysis. The main contribution of the thesis focuses on developing statistically optimal, low-complexity beamforming and RIS-phase shift matrix solution for a RIS-aided downlink MISO system under different propagation environments: Rician-Rician, Rician-Rayleigh, and Rayleigh-Rayleigh. The study considers both correlated and i.i.d. fading scenarios and provides closed-form expressions for optimal transmit beamformers and RIS phase shifts. These solutions offer significant advantages over instantaneous CSI-based beamforming schemes by eliminating the need for reliable feedback channels. The work also derives the ergodic capacity and outage probability performances in closed-form, demonstrating how the number of RIS elements and the presence of LoS components influence system performance under various fading conditions. Next, the thesis investigates beamforming and outage probability analysis for RIS-aided MISO downlink systems under Rician fading along both the direct and the RIS-assisted indirect links. Toward this, the focus has been on maximizing the capacity for two transmitter architectures: fully digital and fully analog. This capacity maximization problem with optimally configured RIS is shown to be L1 norm-maximization with respect to the transmit beamformer. To obtain the optimal fully digital beamformer, a complex L1-PCA-based algorithm has been proposed that has low computational complexity. Another low-complexity optimal beamforming algorithm has been proposed to obtain the fully analog beamforming solution. Additionally, analytical upper bounds on the SNR achievable by the proposed algorithms are derived and used to determine lower bounds on outage probabilities. The derived bounds are numerically shown to be exact for a unit-rank channel matrix. The final part of the thesis explores the design of a RIS-NOMA-aided multi-user downlink system with a low-complexity transceiver architecture. It proposes a fully analog architecture for the base station (BS), using a single radio frequency (RF) chain, which limits it to single-user transmission. To enable multi-user communication, NOMA is integrated with the system. Moreover, the use of RIS enhances the receive SNR for each user and also makes the propagation environment conducive for NOMA, enhancing successive interference cancellation capability. Towards this, the sum rate and energy efficiency of the proposed system are investigated while ensuring minimum rates for all users. Furthermore, the non-convex optimization sum rate and energy efficiency problems are addressed using a quadratic transform-based approach. The proposed RIS-NOMA-aided fully analog system outperforms fully digital architecture in sum rate at lower SNR and energy efficiency across a wide SNR range, demonstrating that RIS can significantly reduce transceiver complexity while improving performance.

Abstract

Imagine a situation where an interior designer is working on a new project and plans to add a few objects in the home to enhance its look. He made all the purchases and now the objects are not fitting into the designed environment. Should he still use them? Instead he can check the suitability beforehand by placing them in the environment virtually and then buy if he is satisfied. Also, consider a scenario in a game where a player is surrounded by enemies approaching from all directions. The player must strategically place obstacles to hold off the enemies, but has no assistance in this task. Can the player survive under these conditions? Can he perform these actions just by voice control or text instruction? Motivated by these ideas, we devised a method to modify the scene based on text prompt in this thesis. Text driven diffusion models have shown remarkable capabilities in editing images. These edits include substituting the objects in scene, inserting new instances, deleting unwanted things and changing the textures. Execution of these edits is quick in images and does not need much guidance. However, when editing 3D scenes, existing works mostly rely on training a Neural Radiance Field (NeRF) for 3D editing. Recent NeRF editing methods leverage edit operations by deploying 2D diffusion models and project these edits into 3D space. They require strong positional priors alongside the text prompt to identify the edit location. In absence of this guidance, edits are decentralized from target location and are diverse in each view. When the changes are different in multiple views, 3D reconstruction becomes inaccurate, changing the overall aesthetics of the scene. These methods are operational on small 3D scenes and are more generalized to a particular scene. They require training for each specific edit and cannot be exploited in real-time edits as each training cycle requires ample amount of time. These limitations make NeRF approach a gridlock for real-time edits. To address these limitations, we propose a novel method, FreeEdit, to make edits in training free manner using mesh representations as a substitute for NeRF. Training-free methods are now a possibility because of the advances in foundation model’s space. We leverage these models to bring a training-free alternative and introduce solutions for insertion, replacement and deletion. We consider insertion, replacement and deletion as basic blocks for performing intricate edits, which are certain combinations of these operations. Given a text prompt and a 3D scene, our model is capable of identifying what object should be inserted/replaced or deleted and the location where the edit should be performed. We also introduce a novel algorithm as part of FreeEdit to find the optimal location on the grounding object for placement. We evaluate our model by comparing it with baseline models on a wide range of scenes using quantitative and qualitative metrics and showcase the merits of our method with respect to others. We hope this work motivates future research on text driven 3D scene edits using mesh representation. As this is an initial step towards training free approach for scene edits, adding few complex components can enhance the user experience that drives the research here after.

Abstract

As cyber threats grow in complexity and sophistication, vulnerabilities are increasingly exploited due to expanding attack surfaces. Knowing all possible vulnerabilities in advance is impractical. Traditional defense techniques often fall short as attackers can conduct thorough reconnaissance on static systems and launch targeted attacks at their convenience. Moving Target Defense (MTD) offers a promising alternative by dynamically altering system configurations, making it harder for attackers to gain a stable understanding of the attack surfaces. However, the implementation of MTD can be costly, with too frequent configuration changes increasing maintenance costs and impacting the quality of service, potentially negating its benefits. Thus, there is a pressing need for cost-efficient MTD strategies that continuously balance robust defense mechanisms with minimal switching overhead. Many existing techniques rely on assumptions about prior knowledge of the attacker’s intentions and payoffs, which limits their applicability in the real-world where such information is often not readily available. This thesis proposes a novel MTD framework designed to not only thwart attacks but also to be cost-aware and continuously adapt to the evolving attack landscape. We leverage the mathematical framework of Factored Markov Decision Process (FMDP) to develop effective defense policies without assuming prior knowledge of the attacker side payoffs. Instead, we incorporate real-time attacker responses into the defender’s MDP using a dynamic Bayesian network. By employing an FMDP, we create a compact yet realistic representation of the system which often consists of multiple changeable aspects that influence the attacker’s response and the rewards. We demonstrate theoretically that considering adaptive attackers in real-world scenarios and devising effective strategies presents inherent challenges, including a negative result on regret bound. However, with certain assumptions, we show that the average regret of our method can be bounded. We validate the efficacy of our method empirically in two domains: a web application and a network system. Our experiments cover various scenarios involving evolving and unknown attackers. The results highlight the potential of our approach in significantly improving defense outcomes and enhancing the scope of modelling complex, real-world scenarios. We include a related co-authored work for comparison, which addresses this problem using a multi-armed bandit (MAB) framework. This approach employs a "followthe-perturbed-leader" style algorithm to learn effective MTD strategies.

Abstract

Software product development can often result in the generation of Software Development Waste (SDW) at any stage of the software development life cycle. SDW is defined as any resource-consuming activity that does not add value to the client or the organization developing the software. SDW impacts a software project’s overall efficiency and productivity as the project scale and size increase. For example, developers may need to rework implementations that cater to ambiguous feature stories; sometimes artifacts may not go into production, resulting in unused artifacts, etc. After the COVID-19 pandemic, software development processes were profoundly impacted. Many organizations are working predominantly with remote or hybrid work models. Traditional practices, reliant on in-person interactions and co-located teams, were disrupted as organizations adapted to new, virtual environments. This transition led to the adoption of various communication and collaboration tools, fundamentally altering how software development was executed and perceived. The work setup required teams to navigate productivity, communication, and workflow management challenges, reshaping established development practices. Our research examined the effects of the pandemic-induced work-from-home situation on the production and handling of SDW. Starting with a multi-year study, we surveyed 615 participants and interviewed 31 from the software industry across eight countries specializing in various domains. We observed a rise in SDWs and identified a new type of SDW, and other wastes were reclassified into existing waste types. Additionally, we found that teams need more direct measures of SDW and rely instead on proxy measures such as productivity and delivery times. The lack of definitive measures to monitor and manage SDW is a concern. The rising adoption of open-source software (OSS) has prompted an examination of the causes of SDW within the use-case of OSS and the appropriate metrics for its measurement. The pandemic reshaped practices, fostering more collaborative and flexible approaches and accelerating OSS adoption. This shift provides a foundation to investigate the influence of these developments on SDW. To address this gap, we propose four measures, namely Stale Forks (SF), Project Diversification Index (PDI), PR Rejection Rate (PRR), Backlog Inversion Index(BII), and the Feature Fulfillment Rate (FFR) visualization to potentially identify unused artifacts, building the wrong feature/product, mismanagement of backlog types of SDW. We applied these measures to ten open-source projects. We observe that OpenCV has 0.85%, a small percentage of active forks, and 95.79%, a high percentage of backup forks, indicating many unused artifacts. Meanwhile, the low stale fork value at 1.43% indicates fewer unused artifacts. Rustlings has a high number of potentially stale (26.11%) and stale forks (10.02%), indicating a high percentage of unused artifacts. The PDI ranges from 0.0143 for Rust in one repository to 2.18 for bootstrap, indicating the rate at which the project’s plan differs from user expectations and how much it aligns with those expectations, respectively. The repository Rustlings has the lowest unused artifact count with a PRR rate of 0.22, while angular and react-native are on the opposite side of the spectrum with values of 6.59 and 19.35, respectively. The project bootstrap has a ’0’ on its BII, with kubernetes showing the highest value at 3.37. Finally, the FFR visualization shows variations in ‘backlog management’ practices among the different projects.