Abstract

In autonomous driving, we have multiple computer vision tasks like object detection, semantic seg- mentation, and instance segmentation which plays a crucial role in perceiving the environment around the vehicle. Understanding the behaviour and performance of such tasks helps improve and address the key issues that are inherent in the system. There can be issues which are latent in the deep learning architecture and also in the datasets on which the deep learning models are trained and tested. In this thesis, we benchmark the performance of various popular deep learning models on road scene datasets for various computer vision tasks and also formulate open-world object detection on road scenes by addressing the inherent issues present in road scene datasets. In the first part of the work, we aim to understand the performance and behaviour of various deep learning models on road scene datasets; Cityscapes, IDD, and BDD. Object detection, semantic seg- mentation, and instance segmentation form the bases for many computer vision tasks in autonomous driving. The complexity of these tasks increases as we shift from object detection to instance seg- mentation. The state-of-the-art models are evaluated on standard datasets such as PASCAL-VOC and MS-COCO, which does not consider the dynamics of road scenes. Driving datasets such as Cityscapes and Berkeley Deep Drive(BDD) are captured in a structured environment with better road markings and fewer variations in the appearance of objects and background. However, the same does not hold for Indian roads. The Indian Driving Dataset(IDD) dataset is captured in unstructured driving scenarios and is highly challenging for a model due to its diversity. This work presents a comprehensive evaluation of state-of-the-art models on object detection, semantic segmentation, and instance segmentation on road scene datasets. We present our analyses and compare their quantitative and qualitative performance on structured driving datasets(Cityscapes and BDD) and the unstructured driving dataset(IDD); under- standing the behavior on these datasets helps in addressing various practical issues and helps in creating real-life applications. In the second part of the work, we model open-world object detection on road scenes since object detection is a crucial component in autonomous navigation systems. Current object detectors are trained and tested on a fixed number of known classes. However, in real-world or open-world settings, the test set may consist of objects of unknown classes; this results in the unknown objects being falsely de- tected as known objects leading to the failure in the decision making of autonomous navigation systems. We propose Open World Object Detection on Road Scenes (ORDER) to resolve the aforementioned problem. We introduce Feature-Mix that widens the gap between known and unknown classes in latent feature space and improves the unknown object detection in the ORDER framework. We identify the inherent problems present in autonomous datasets: i) a significant proportion of the dataset comprises small objects and ii) intra-class bounding box scale variations. We address the problem of small object detection and intra-class bounding box variations by proposing a novel focal regression loss. Further, the detection of small objects is improved by curriculum learning. We present an extensive evaluation of two road scene datasets: BDD and IDD. Our experimental evaluations on BDD and IDD shows con- sistent improvement over the current state-of-the-art method. We believe that this work will lay the foundation for real-world object detection for road scenes.

Abstract

In spite of the vast work in the area of Maximal Frequent Subgraph (MFS) mining, current algorithms do not scale to large databases with numerous graphs. Even the smallest search space required by current MFS algorithms for large databases is too huge to work within the main memory alone, making the algorithms I/O bound. Added to this is the expensive cost of subgraph isomorphism operation, which is commonly used for frequency computation. This thesis explores partition-based secondary memory algorithms and proposes Partition-based Frequent Subgraph Mining using Maximum Common Subgraph (PMCS), that can make maximal frequent subgraph mining for large databases viable. The naive application of the partition-based approach is inefficient due to the need to recompute subgraph frequency and the possibility of storing intermediate subgraphs on the disk. In PMCS, we overcome these drawbacks by intelligently avoiding redundant computations and performing frequency computation operations optimally using the Maximum Common Subgraph operation. Our intermediate results are also small enough to fit in the main memory. While existing algorithms fail to scale beyond 300k, our results demonstrate that PMCS scales to databases of up to 1000k graphs with an average of 25-30 edges per graph.

Abstract

Spoken utterances have an inherent structure in the sense that some words seem to group naturally together and some words seem to have a notable break or disjuncture between them. This can be described in terms of prosodic phrasing, meaning that a spoken utterance has a prosodic phrase structure, similar to how a written utterance has a syntactic phrase structure. Phrase breaks in natural speech are important; they are physiologically essential, help emphasize content, and improve the intelligibility of speech. The process of inserting phrase breaks in an utterance is called phrasing. In the context of speech synthesis, phrasing is a crucial step. It breaks long utterances into meaningful units of information and improves the intelligibilty of the synthesized speech. Traditional methods of phrase break prediction have used discrete linguistic resources like part-ofspeech (POS) sequence information for modeling these breaks. These methods cannot be used for languages where the necessary linguistic resources are not readily available, resulting in unsupervised methods of inducing word representations which can be used as surrogates for POS tags in phrase break prediction. However, these methods are not suitable in the context of Indian languages, which are aggutinative in nature resulting in an increase in vocabulary size. This thesis presents the use of word terminal syllables (last syllable of the word) to model phrase breaks. We demonstrate the correlation between these terminal syllables and acoustic breaks found in the speech signal. We utilize these terminal syllables in building models for phrase break prediction from text in six Indian languages and demonstrate by means of objective and subjective measures that these models perform as well as traditional models using POS sequence information. The discrete linguistic representations (like POS tags, induced POS tags, word-terminal syllables) used by traditional methods of phrase break prediction require a hard classification of words into a set of predefined discrete classes, which raises issues when there is ambiguity in the linguistic representation of a word. Moreover, such a representation does not capture the co-occurrence statistics of the words, i. e., they do not take into account the distributional behaviour of words. Both these issues can be addressed by the use of continuous dimensional representations of words also known as word embeddings. This thesis presents a neural network architecture to induce task specific word embeddings for phrase break prediction. We show that our proposed architecture combines feature induction and phrase break prediction in a single framework and thus avoids the two stage process required while using features derived from Latent Semantic Analysis (LSA). We train our model on audiobook data and show that these task specific word features are better word representations as compared to those derived using LSA, for phrase break prediction. With the advent of deep learning, there have been attempts to apply deep neural networks (DNNs) to phrase break prediction. While deep neural networks are able to effectively capture dependencies across features, they lack the ability to capture long-term relations that are spread over time. On the other hand, recurrent neural networks (RNNs) are able to capture long-term temporal relations and thus are better suited for tasks where sequences have to be modeled. This thesis presents the use of RNNs for phrase break prediction, and shows by means of experimental results that they perform better than DNNs. Traditional techniques for learning word embddings learn static or context-free word embeddings, meaning that for each word in the vocabulary, one fixed embedding is learnt for all possible contexts in which that particular word appears in the text corpus. While static embeddings have proved to be effective in NLP applications, they do not take into account the fact that words have different meanings in different contexts. To address this issue, context-sensitive word representations, also called dynamic or contextual word embeddings were developed. The most popular technique for learning dynamic word embeddings is BERT (Bidirectional Encoder Representations from Transformers). This thesis presents work on phrase break prediction using bidirectional encoder representations learnt from fine-tuning a pretrained BERT model, with an additional token classification layer, on phrase break prediction. We show that representations using this model outperform task-specific static word embeddings learnt using a BLSTM token classification model trained from scratch. Finally, this thesis presents work which attempts to answer the following questions: (a) Is there any utility in incorporating an explicit external phrasing model in an End-to-End TTS system? and (b) How do you evaluate the effectiveness of a phrasing model in an End-to-End TTS system?

Abstract

Ancient manuscripts were among the first forms of written communication, offering key insights into our past, covering literature, medicine, culture, philosophy, religion, and more. It is imperative to save these writings to identify and extract their hidden knowledge. Document collections frequently exhibit overlapping components, irregular patterns, dense layouts, extremely high aspect ratios, physical and chemical degradation (evident in ink-based manuscripts), text misalignment, and more. Compounding these difficulties are issues that may arise during the digitization processes, such as improper document positioning, inadequate illumination, and scanner effects. In this academic thesis, our main emphasis is on identifying and segmenting text lines inside these documents for downstream OCR applications with utmost precision. We devise a two-stage approach - SeamFormer- to identify special text baselines in palm-leaf manuscripts using a multi-task strategy via Vision Transformers (ViTs). In the first stage, we detect text strikethroughs, namely ’scribbles,’ which act as pointers to the location of text line segments within the document. The encoder-decoder architecture is used to analyze input image patches and produce two separate maps: a scribble map and a binary map. In the second stage, we post-process the prior stage outputs and generate a diacritic-aware global energy map. To generate the final precise text line polygons, we use a modified seam generation algorithm along with customized global maps. The prior methodology is further enhanced by the proposed LineTR model, a multi-task DETR (Detection Transformer) model that reconceptualizes the scribble generation as a geometric problem. This method simply generates line parameters for each text line present in the input image patch. This design decision enables the model to exhibit zero-shot behavior across diverse historical manuscripts. The state-of-the-art approach has been proven to generate precise text-line segmentation with a single ’unified’ model with minimal post-processing efforts, making it a strong candidate for image-to-OCR integration pipelines

Abstract

The surge in robotics, IoT, machine vision, biometrics, security cameras, and smart devices has led to a rising demand for image sensors and cameras. Complementary Metal-Oxide-Semiconductor (CMOS) light sensors have become the heart of modern digital imaging, widely utilized in diverse applications ranging from consumer electronics to industrial and scientific domains, especially in highly sensitive areas, low luminescent conditions, labs on chips, etc. This thesis centers around the CMOS light sensors, discussing their architecture, performance, and integration. The research mainly focuses on CMOS-compatible photodiodes and single-photon avalanche diodes (SPADs). For the CMOS photodiodes, this work’s primary idea is to enhance their light sensitivity and photon collection efficiency and integrate these photodiodes with the circuits based on standard CMOS processes. This involves the ability to modify the structure or shape of the photodiode in order to increase the efficiency of photon collection when the photodetector is in use. This research work tackles low-light imaging by using single-photon avalanche diode detectors, which efficiently identify single photons. Another important consideration when using SPAD in such an application is using a high-speed quenching circuit to reduce afterpulsing. Different SPAD architecture simulation models are reviewed and compared to model the SPAD accurately. In this work, an active quenching circuit is designed to facilitate fast quenching and minimal dead time.The work encompasses extensive simulation and modeling using tools such as Cadence Virtuoso and COMSOL Multiphysics, allowing for precise evaluation of the proposed design’s performance metrics. The results show significant enhancements in sensor performance and the possibility of expanding its use in biomedical imaging applications, robotics, and other sensitive imaging needs.

Abstract

A Completely Automated Public Turing Test to Tell Computers and Humans Apart (CAPTCHA) is one of the primary barriers between notorious bots and legitimate human users. However, advancements in Artificial Intelligence (AI) have enabled malicious bots to circumvent CAPTCHA challenges effectively. As a result, several types of CAPTCHA have been rendered ineffective. In this work, we introduce Swiss Cheese CAPTCHA, a novel sensor-based solution designed to be easily solvable by humans while presenting multiple obstructions for bots (similar to the Swiss Cheese Model) even when the sensor outputs can be predicted and interfered with. We leverage a range of human cognitive abilities and Generic Sensor API in modern devices to provide robust protection against automated attacks by making it more computationally expensive for bots to produce a valid answer within a stipulated time. We conducted two user studies to assess our proposal’s effectiveness: one involving 116 participants to assess the likability and improvise the design, and the other, with 107 participants, to investigate the impact of improvised design changes on cognitive abilities. Our results from these studies show an average completion time of 4.76 seconds and 6.12 seconds, with a success rate of 90.3% and 83.25%, respectively. By analyzing the 2141 resultant trajectories from both user studies, we assess the learnability, error recovery rate, efficiency, and satisfaction of users using the scheme. Finally, we devise an automated attack against our proposal to analyze its security in the real world; we find the probability of attack success is low. We also make our dataset available for further research.

Abstract

This thesis presents a comprehensive exploration of Multi-Object Multi-Part Scene Parsing in 2D images, showcasing significant advancements through two novel approaches, FLOAT and OLAF, each tailored to enhance scene parsing performance and scalability. The first paper introduces FLOAT, a factorized label space framework designed to independently predict object categories and part attributes, thereby simplifying the segmentation task and enhancing scalability. Notably, FLOAT incorporates a unique ’zoom’ refinement technique at inference time, significantly elevating segmentation accuracy, particularly for smaller objects and parts. Empirical results on the Pascal-Part datasets underscore FLOAT’s superior performance, achieving notable improvements in mean Intersection Over Union (mIOU) and segmentation quality IOU (sqIOU), especially on the most comprehensive PascalPart-201 dataset, reflecting its effectiveness in handling diverse and complex scenes. The second paper delves into OLAF, a plug-and-play methodology that augments traditional RGB inputs with object-based structural cues to better capture the complexities of scene structures. This approach leverages a weight adaptation technique, allowing pre-trained RGB models to seamlessly integrate augmented data, thus stabilizing the optimization process. Additionally, the introduction of the LDF encoder module aids in providing low-level dense feature guidance, enhancing the segmentation of smaller parts. OLAF demonstrates its versatility across various architectures and datasets, achieving significant mIOU gains on multiple Pascal-Part benchmarks, highlighting its broad applicability and robust performance enhancements in challenging segmentation scenarios.

Abstract

In the era of big data, distributed computing frameworks like Hadoop’s MapReduce have become essential for processing massive datasets across clusters of servers. A critical phase in the MapReduce paradigm is the shuffling stage, where intermediate data produced by Map tasks is transferred to the appropriate Reduce tasks. This data shuffling often incurs significant communication overhead, leading to performance bottlenecks and increased job completion times. A key factor influencing communication overhead is data locality, which refers to the practice of processing data close to its storage location. By minimizing the distance data must travel across a network, data locality reduces latency, conserves bandwidth, and improves overall system performance. To address these challenges, the Coded MapReduce framework was introduced, using coding techniques to create multicast opportunities during the shuffling phase. By replicating Map tasks across multiple servers, this approach effectively reduces inter-server communication load, enhancing overall computational efficiency. Ho, Wu, and Liu demonstrated that optimizing reducer placement to reduce cross-rack traffic can lead to significant performance improvements, achieving up to a 32% enhancement in job execution times. Building upon this foundation, our research presents the Hybrid Coded MapReduce scheme, tailored for server–rack architectures prevalent in contemporary data centers. In such configurations, servers are organized into racks, with intra-rack communication being faster and more efficient than cross-rack communication. Recognizing this disparity, our Hybrid Coded MapReduce approach aims to minimize costly cross-rack data transfers while optimizing data locality. Central to our methodology is the formulation of the Map task assignment problem as a constrained integer optimization problem. This formulation seeks to assign Map tasks to servers in a manner that maximizes data locality, ensuring data is processed in proximity to its storage location. Consequently, the need for cross-rack communication is reduced, leading to decreased overall communication costs and improved job completion times. To validate our approach, we conducted extensive simulations comparing the performance of the Hybrid Coded MapReduce scheme against the traditional Coded MapReduce framework. Our results demonstrate that the Hybrid approach significantly reduces cross-rack communication without compromising the benefits of coded multicasting. Furthermore, by optimizing data locality, our scheme improves data processing efficiency, resulting in faster job completion and better resource utilization. In conclusion, the Hybrid Coded MapReduce scheme offers a practical and effective solution to improve the performance of distributed computing systems within server–rack architectures. By intelligently balancing the trade-offs between communication cost and data locality, our approach paves the way for more efficient and scalable big data processing.

Abstract

In Indian urban and rural driving scenarios, small objects are pervasive and often crucial for safe navigation. These objects can include pedestrians crossing roads, children playing near streets, cyclists, stray animals, as well as small vehicles like scooters and motorbikes. Additionally, traffic signs, signal lights, potholes, and road markings (such as lane dividers or zebra crossings) are often small in size but essential for driving decisions. In such contexts, missing or inaccurately segmenting these small objects can lead to critical errors in detection, causing accidents or delays in the vehicle’s decision-making process. Automated understanding of such objects need detection and segmentation to start with. Semantic Segmentation is a critical task in computer vision with a wide range of applications. The objective is to partition an image—a collection of pixels—into distinct labeled regions, each corresponding to specific objects or parts of the scene. This process is crucial for scene understanding and enables the localization of objects within the image. Over time, significant progress has been made in semantic segmentation, especially with the advent of deep learning. The advances in this area have revolutionized computer vision, pushing beyond traditional methods and achieving remarkable improvements in performance. When discussing semantic segmentation, we often focus on datasets, the objects within those datasets, and their corresponding segmentations. While many datasets exist for road scenarios, particularly those representing Western road conditions, there is relatively little research on road conditions specific to India. One notable exception is the Indian Driving Dataset (IDD), a dataset specifically designed for semantic segmentation of Indian road scenarios. Road and driving datasets typically contain objects of varying sizes within each class label. These objects can be broadly categorized into three types: small, medium, and large. The importance of segmentation is well understood across several domains such as medical imaging, autonomous vehicles, aerial imagery, robotics, surveillance, and industrial automation. However, one of the most challenging problems in segmentation is the segmentation of small objects. Small object segmentation is particularly difficult due to factors such as (i) the limited number of pixels representing small objects, (ii) class imbalance during training, and (iii) the inherent challenges posed by small object representations. These factors hinder the performance of deep learning architectures, making it harder for modern techniques to accurately handle small objects. This issue becomes evident when evaluating deep learning models, as performance metrics tend to be significantly worse for small objects. In this study, we delve into the challenges of segmenting small objects and explore potential strategies for improving algorithm design for this task. It is widely acknowledged that loss functions play a key role in segmentation performance, alongside the architecture of deep learning models. Indeed, the segmentation of small objects is highly influenced by the choice of loss function during model training. Various small objects commonly encountered in road scenarios are highlighted in this study, emphasizing their importance in Indian road environments. Objects such as pedestrians, traffic signs, and traffic lights—despite their small size—are crucial for tasks like autonomous navigation and driver assistance systems. The Indian Driving Dataset (IDD) offers a valuable resource, containing a wide range of small objects across different class labels, some of which dominate specific categories. These objects are captured and annotated in real-world, unstructured environments, making this dataset particularly relevant for segmentation tasks in Indian road scenarios. Therefore, our focus in this study is primarily on detection and segmentation of small objects within the IDD dataset and explore the impact of loss functions.

Abstract

Principal component analysis(PCA) is one of the most popular linear dimensionality reduction techniques in machine learning. In this thesis, we present a method for performing PCA on encrypted data using a homomorphic encryption scheme. We used the CKKS homomorphic encryption scheme for our purpose since it is most suitable for machine learning tasks allowing approximate computations on real numbers. In this thesis, we propose a new Homomorphic PCA(HPCA) algorithm that performs PCA. Unlike previously proposed algorithms, our HPCA algorithm is non-interactive in nature. This requires an approximation of the inverse sqrt function. This thesis presents two methods to approximate and securely perform the inverse sqrt function using CKKS homomorphic encryption scheme - Constrained Linear Regression and Pivot-Tangent method. Since the CKKS homomorphic scheme allows only the computation of polynomial functions, we propose a method to approximate the inverse sqrt function polynomially. Ultimately, we implement our approach for the inverse sqrt function. We provide an implementation of our method for the inverse sqrt function. We use the inverse sqrt approximations in our HPCA algorithm to make it non-interactive. In the end, we present the experimental results of our proposed HPCA algorithm on a few datasets computing a few principal components. We measure the R2 score on the reconstructed data and use it as an evaluation metric for our HPCA algorithm.