Abstract

Ancient paper documents and palm leaf manuscripts from the Indian subcontinent have made a significant contribution to the world literary and culture. These documents often have complex, uneven, and irregular layouts. The process of digitization and deciphering the content from these documents without human intervention pose difficulties in a broad range of areas, including language, script, layout, elements, position, and number of manuscripts per image. Large-scale annotated Indic manuscript image datasets are needed for this kind of research. In order to meet this objective, we present Indiscapes, the first dataset containing multi-regional layout annotations for ancient Indian manuscripts. We also adapt a fully convolutional deep neural network architecture for fully automatic, instance-level spatial layout parsing of manuscript images in order to deal with the challenges such as presence of dense, irregular layout elements, pictures, multiple documents per image and the wide variety of scripts. Eventually, We demonstrate the effectiveness of proposed architecture on images from the Indiscapes dataset. Despite advancements, the segmentation of semantic layout using typical deep network methods is not resistant to the complex deformations that are observed across semantic regions. This problem is particularly evident in the domain of Indian palm-leaf manuscripts, which has limited resources. Therefore, we present Indiscapes2, a new expansive dataset of various Indic manuscripts with semantic layout annotations, to help address the issue. Indiscapes2 is 150% larger than Indiscapes and contains materials from four different historical collections. In addition, we propose a novel deep network called Palmira for reliable, deformation-aware region segmentation in handwritten manuscripts. As a performance metric, we additionally report a boundary-centric measure called Hausdorff distance and its variations. Our tests show that Palmira offers reliable layouts and outperforms both strong baseline methods and ablative versions. We also highlight our results on Arabic, South-East Asian and Hebrew historical manuscripts to showcase the generalization capability of PALMIRA. Even though we have reliable deep-network based approaches for comprehending manuscript layout, these models implicitly assume one or two manuscripts per image during the process, whereas in a real-world scenario there are often cases where multiple manuscripts are typically scanned together into a scanned image to maximise scanner surface area and reduce manual labour. Now, making sure that each individual manuscript within a scanned image can be isolated (segmented) on a per-instance basis became the first essential step in understanding the content of a manuscript. Hence, there is a need for a precursor system which extracts individual manuscripts before downstream processing. The highly curved and deformed boundaries of manuscripts, which frequently cause them to overlap with each other, introduce another complexity when confronting issue. We introduce another new document image dataset named IMMI (Indic Multi Manuscript Images) to address these issues. We also present a method that generates synthetic images to augment sourced non-synthetic images in order to boost the efficiency of the dataset and facilitate deep network training. Adapted versions of current document instance segmentation frameworks are used in our experiments. The results demonstrate the efficacy of the new frameworks for the task. Overall, our contributions enable robust extraction of individual historical manuscript pages. This in turn, could potentially enable better performance on downstream tasks such as region-level instance segmentation, optical character recognition and word-spotting in historical Indic manuscripts at scale.

Abstract

Brain decoding involves the reconstruction of stimuli from brain recordings. These recordings can be obtained by presenting stimuli to a subject in various forms, such as text, image, and speech. Despite extensive research on brain decoding, important questions remain unanswered. Can we develop multi-view decoders capable of decoding concepts from brain recordings of any view, including picture, sentence, or word cloud? Can we build a system that can use brain recordings to automatically generate descriptions of what a subject is viewing using keywords or sentences? How about a system that can automatically extract important keywords from sentences that a subject is reading? Answering these questions requires innovative approaches to brain decoding, as traditional methods have not yet been proven adequate. Previous brain decoding efforts have focused only on single-view analysis and hence cannot help us build such systems. As a first step toward building such systems, inspired by Natural Language Processing literature on multi-lingual and cross-lingual modelling, this thesis proposes novel brain decoding setups: (1) Multi-view Decoding (MVD), (2) Cross-view Decoding (CVD), and (3) Abstract v/s Concrete Decoding. In MVD, the goal is to build an MV decoder that can take brain recordings for any view as input and predict the concept. In CVD, the goal is to train a model which takes brain recordings for one view as input and decodes a semantic vector representation of another view. Specifically, this thesis studies practically useful CVD tasks like image captioning, image tagging, keyword extraction, and sentence formation. In Abstract v/s Concrete Decoding, the goal is to build a decoder trained on concrete concepts and test it on both abstract and concrete concepts and similarly build a decoder trained on abstract concepts and test it in both types of concepts. Extensive experiments lead to MVD models with ∼0.68 average pairwise accuracy across view pairs and CVD models with ∼0.8 average pairwise accuracy across tasks. It was found that the decoder trained on concrete concepts can decode both abstract and concrete objects with great and better accuracy than the model trained on abstract objects. Analysis of the contribution of different brain networks reveals exciting cognitive insights: (1) Models trained on picture or sentence view of stimuli are better MV decoders than a model trained on word cloud view. (2) Our extensive analysis across 9 broad brain regions, 11 language sub-regions, and 16 visual sub-regions of the brain helped us localize, for the first time, the parts of the brain involved in cross-view tasks like image captioning, image tagging, sentence formation, and keyword extraction. (3) The visual brain network is very important for processing Word+Picture stimuli for concrete concepts. Surprisingly, this is not the case for abstract concepts where voxels from the language and DMN brain network are more activated.

Abstract

The task of saliency prediction focuses on understanding and modeling human visual attention (HVA), i.e., where and what people pay attention to given visual stimuli. Audio has ideal properties to aid gaze fixation while viewing a scene. There exists substantial evidence of audio-visual interplay in human perception, and it is agreed that they jointly guide our visual attention. Learning computational models for saliency estimation is an effort to inch machines/robots closer to human cognitive abilities. The task of saliency prediction is helpful in many digital content-based applications like automated editing, perceptual video coding, human-robot interactions,etc. The field has progressed from using hand-crafted features to deep learning-based solutions. Efforts on static image saliency prediction methods are led by convolutional architectures. The ideas were extended to videos by integrating temporal information using 3D convolutions or LSTM’s. Many sophisticated multimodal, multi-stream architectures have been proposed to process multimodal information for saliency prediction. Despite existing works of Audio-Visual Saliency Prediction (AVSP) models claiming to achieve promising results by fusing audio modality over visual-only models, most of these models only consider visual cues and fail to leverage auditory information that is ubiquitous in dynamic scenes. In this thesis, we investigate the relevance of audio cues in conjunction with the visual ones and conduct extensive analysis to analyse the cause of AVSP models being superior by employing well-established audio modules and fusion techniques from diverse correlated audio-visual tasks. Our analysis on ten diverse saliency datasets suggests that none of the methods worked for incorporating audio. Our endeavour suggests that augmenting audio features ends up learning a predictive model agnostic to audio . Furthermore, we bring to light, why AVSP models show a gain in performance over visual-only models, though the audio branch is agnostic at inference. Our experiments clearly indicate that visual modality dominates the learning; the current models largely ignore the audio information. The observation is consistent while using three different audio backbones and four different fusion techniques and contrasts with the previous methods, which claim audio as a significant contributing factor. The performance gains are a byproduct of improved training and the additional audio branch seems to have a regularizing effect. We show that similar gains are achieved while sending random audio during training. Overall our work questions the role of audio in current deep AVSP models and motivates the community to a clear avenue for reconsideration of the complex architectures by demonstrating that simpler alternatives work equally well.

Abstract

Detecting humans in images and videos has emerged as an essential aspect of intelligent video systems that solve pedestrian detection, tracking, crowd counting, etc. It has many real-life applications varying from visual surveillance and sports to autonomous driving. Despite achieving high performance, the single camera-based detection methods are susceptible to occlusions caused by humans, which drastically degrades the performance where crowd density is very high. Therefore multi-camera setup becomes necessary, which incorporates multiple camera views for detections by computing precise 3D locations that can be visualized and transformed to Top View also termed as Bird’s Eye View (BEV) representation and thus permits better occlusion reasoning in crowded scenes. The thesis, therefore, presents a multi-camera approach that globally aggregates the multi-view cues for detection and alleviates the impact of occlusions in a crowded environment. But it was still primarily unknown how satisfactorily the multi-view detectors generalize to unseen data. In different camera setups, this becomes critical because a practical multi-view detector should be usable in scenarios such as i) when the model trained with few camera views is deployed, and one of the cameras fails during testing/inference or when we add more camera views to the existing setup, ii) when we change the camera positions in the same environment and finally iii) when deploying the system on the unseen environment; an ideal multi-camera setup system should be adaptable to such changing conditions. While recent works using deep learning have made significant advances in the field, they have overlooked the generalization aspect, which makes them impractical for real-world deployment. We formalized three critical forms of generalization and outlined the experiments to evaluate them: generalization with i) a varying number of cameras, ii) varying camera positions, and finally, iii) to new scenes. We discover that existing state-of-the-art models show poor generalization by overfitting to a single scene and camera configuration. To address the concerns: (a) we generated a novel Generalized MVD (GMVD) dataset, assimilating diverse scenes with changing daytime, camera configurations, varying number of cameras, and (b) we discuss the properties essential to bring generalization to MVD and developed a barebones model to incorporate them. We performed a series of experiments on the WildTrack, MultiViewX, and the GMVD datasets to motivate the necessity to evaluate the generalization abilities of MVD methods and to demonstrate the efficacy of the developed approach.

Abstract

Space, time, and number are fundamental to human cognition. The mental representations on these entities are crucial for planning and decision-making. In our everyday lives, we are always thinking in terms of quantities — how long would we take to reach the workplace, what would be a shorter route to get to a specific store from where we are, how many cupcakes should we prepare for the people we have invited, how do we throw a stone that will dislodge a shuttlecock stuck in the tree, and so on. Even for simple tasks like grasping, reaching, or catching a ball, subtle calculations involving distance, speed, and time are essential. To successfully execute our actions, we need to synchronize these entities efficiently. For example, to grab an object kept on the table, one needs to integrate information from time, space, and number dimensions to evaluate the obstacles present in that environment and the distance between the object and our body. Further, spatiotemporal integration of information is needed for successful reaching and grasping. Over the last two decades, numerous studies have advanced our knowledge of how humans utilize perceptual information to estimate magnitudes such as space, time, and number. One of the most popular theories of magnitude processing, A Theory of Magnitude (ATOM), suggests that a generalized magnitude system in the brain processes information related to space, time, and numbers. Since these magnitudes are processed by a common magnitude system, they interact with one another. Earlier studies investigating the number-time interaction have provided support to ATOM’s predictions. On the contrary, more recent studies have argued against ATOM and suggested that the cross-dimensional magnitude interactions may emerge from cognitive factors like attention and memory. Such contradicting findings raise a fundamental question as to whether a common magnitude system indeed exists, or such crossdimensional magnitude interactions result from cognitive factors. This is still an unsettled question. In the present thesis, we examine the influence of numerical magnitude on temporal processing in a different experimental setup. More specifically, we investigated whether numerical magnitude affects our temporal experience or simply biases judgment of time. Further, we examined whether large numerical magnitude is always perceived to be longer in time (as predicted by ATOM) or attentional modulation can cause such crossdimensional magnitude interactions. We also tested the generality of the ATOM framework in a cross-modal setting wherein numerical magnitude and temporal information were presented in two different modalities and evaluated ATOM’s prediction in a cross-domain setting. The overall results from the five empirical investigations suggest that the processing of numbers and time may not require to invoke a common magnitude processing system. The cross-dimensional magnitude interactions (in this case, number and time) may emerge from the modulation of attentional mechanisms.

Abstract

Retail stores have become a part of our daily lives and play a vital economic role in society. It has been reported that shelf-space allocation decisions in a retail store significantly impact the retailer’s revenue. So, developing approaches for efficient product placement in available shelf space in retail stores is one of the key research issues. Several research efforts have been made to propose approaches for improving product placement in retail stores based on the knowledge of customer purchase history. Traditionally, several dynamic programming model based approaches were proposed to solve this problem. More recently, data mining approaches, like frequent pattern mining and utility pattern mining, have been employed to improve product placement based on the patterns extracted from customer purchase transactions. In this thesis, we address the issue of product placement in a retail store by considering the diversity of products. Providing diverse options to customers can be useful in increasing the long term sustainability of retail stores. Other recommendation systems have also used increasing diversity to improve user interest. In the context of retail stores, a store that can cater to the needs of a diverse customer base (by providing diverse recommendations) has a higher chance to stay relevant for customers in the long term, thus, facilitating long term sustainability of retail businesses. We have proposed an approach for facilitating the placement of items in a diversified manner in a given retail store based on the concept hierarchy that exists among the items without compromising the revenue. In the proposed approach, a concept hierarchy based approach is employed to determine the diversity value of the itemset quantitatively. The diversity of the given itemset captures the extent of its items belonging to multiple categories. By combining the notion of utility and diversity, we propose a framework to compute diverse net revenue of the itemset. Given a set of transactions, price values of items, and concept hierarchy, we propose a methodology to build the Concept Hierarchy Utility Itemset Index (CHUI), which contains potential itemsets with high revenue and diversity. Next, we propose the approach to perform product placement based on the knowledge of itemsets in the CHUI. We conduct experiments on a real dataset, namely, Instacart retail dataset (containing 49,688 items and 1,31,208 transactions), to demonstrate the proposed approach’s overall effectiveness

Abstract

Rice is a fascinating and complex plant. Consumed by more than half of the world’s population, it is important to have a comprehensive understanding of the organism to advance crop engineering and breeding strategies. Abiotic stresses like drought, high temperature, salinity and flood have affected its growth and productivity. Furthermore, global climate change has added to the severity of these stresses, suggesting the need for varieties with improved stress tolerance for sustainable crop production. Improving stress tolerance requires an in-depth understanding of the biological processes, transcriptional pathways and hormone signaling involved in stress response. With the surge in omics data, it has paved the way for deciphering the biological information underlying complex traits. However, dealing with such large datasets calls for the development of powerful bioinformatics methods for a thorough transcriptome analysis. A popular approach is the construction and analysis of co-expression networks representing transcriptionally coordinated genes that are often part of the same biological process. Using prior knowledge and data integration further enhances the elucidation of gene regulatory relationships in this network. With this objective we have developed NetREx, a Network based Rice Expression Analysis Server, that hosts ranked co-expression networks of Oryza sativa using publicly available mRNAseq data across uniform experimental conditions. It provides a range of interactable data viewers and modules for analysing user queried genes across different stress conditions (drought, flood, cold and osmosis) and hormonal treatments (abscisic and jasmonic acid) and tissues (root and shoot). Subnetworks of user-defined genes can be queried in preconstructed tissue-specific networks, allowing users to view the fold-change, module memberships, gene annotations and analysis of their neighborhood genes and associated pathways. The webserver also allows querying of orthologous from Arabidopsis, wheat, maize, barley, and sorghum. Here we demonstrate that NetREx can be used to identify novel candidate genes and tissue-specific interactions under stress conditions and can aid in the analysis and understanding of complex phenotypes linked to stress response in rice. Available at: https://bioinf.iiit.ac.in/netrex/. In the second part of the thesis, we present a meta-analytic study using co-expressed modules to understand the biological functions associated with different abiotic stresses in the root tissue. The osmotic stress condition is an extremely severe stress condition involving the effects of multiple stresses like drought, salinity and ionic stress and is discussed in detail. The early responsive modules are analyzed and a causal flow of mechanisms and signaling pathways is established.

Abstract

Typically, anatomy is taught through dissection, 2D images, presentations, and cross-sections. While these methods are convenient, they are non-interactive and fail to capture the spatial relationships and functional aspects of anatomy. New visualization technologies such as virtual reality and 3D can compensate for the impediments and provide better understanding while captivating the students. With recent advances in the industry, the methods to provide a 3D experience are economical. In this thesis, we introduce a low-cost 3D-interactive anatomy system designed for an audience of typical medical college students in an anatomy class. The setup used to achieve 3D visualization is Dual projector polarization. While there are other ways to achieve 3D visualization, like alternate frame sequencing and virtual reality, this technique can target a large audience and requires minimum accessories for the setup enabling this to be a low-cost solution for an immersive 3D experience. The 3D interactive Neuroanatomy solution is an end-to-end framework capable of designing anatomy lessons and visualizing the 3D stereoscopic projection of those anatomy lessons. To ensure superior comprehension of students, we incorporate each teacher’s unique teaching approach while developing anatomy lessons by providing the ability to create their own lessons. We have created anatomy lessons based on the human brain which is a vital organ and has a complex anatomy. Our aim is to help medical students to understand the complexity of organ systems from not just an anatomical perspective but also a radiological perspective. We use annotations on clinical case data such as MRI, MRA, etc., to create 3D models for anatomy visualization incorporating clinical information and illustrating real cases. Annotations for structures of interest are done using manual, automatic, and semi-automatic segmentation methods. Manual delineation of the structure boundaries is very tedious and time-consuming. Automatic segmentation is quick and convenient. However, manual annotations were done for the 3D anatomy viewer for small and complex structures due to substandard automatic segmentation. There is a need to improve automatic segmentation performance for those structures. While segmentation is an essential step in 3D modeling, it plays a critical role in many neurological disease diagnoses as well, which are associated with degradation in the sub-cortical region. Therefore accurate algorithms are needed for sub-cortical structure segmentation. Variance in the size of structures is significant, which introduces a performance bias towards larger structures in many deep learning approaches. In this part of the thesis, we aim to remove size bias in sub-cortical structure segmentation. The proposed method addresses this problem with a pre-training step used to learn tissue characteristics, an ROI extraction step that aids in focusing on the local context, and using structure ROIs elevates the influence of smaller structures in the network.

Abstract

FPGAs are increasingly significant for deploying convolutional neural network (CNN) inference models because of performance demands and power constraints in embedded and data centre applications. The compute intensity of these models makes prototyping highly complex and time-consuming with traditional RTL approaches. The release of new generation high-level synthesis tools (HLS), such as Intel FPGA SDK for OpenCL, and Xilinx’s VITIS Unified Software Development platform, have significantly reduced the time and complexity of prototyping complex designs on FPGA. This work involves building custom FPGA accelerators for image recognition systems using OpenCL-HLS. Object detection and classification are vital steps in building image recognition systems. The first part of the work concerns building an FPGA accelerator for the Traffic Sign Classification problem, a vital step in building traffic sign recognition (TSR) systems that employ vehicle-mounted cameras that identify traffic signs while driving on the road. However, the CNNs for the classification still need the ability to be spatially invariant to the input data. Spatial Transformers are learnable modules that, upon integration with CNN, would allow the spatial manipulation of data within the network, making it invariant to affine transformations. Generic Matrix multiply (GEMM) methods that express convolution as matrix multiplication are widely used in deep-learning frameworks like Caffe, Theano, and Torch with GPU support. im2row is one of the commonly used GEMM methods. In this work, we built a GEMM-based accelerator for a CNN with a Spatial transformer module. We proposed the channel adaptive im2row method, with a lesser on-chip memory footprint than im2row. The system attains a latency of 202 ms (5˜ fps), running at 202 MHz on Intel Arria10 GX FPGA, and attains a speedup of (> 5 X) compared to the CPU. The performance is not state-of-the-art and calls for more FPGA-specific optimizations. Further, from the learnings, we designed a Systolic Array accelerator with a novel load pattern for accelerating the widely-used object detector YOLOv3-tiny optimized explicitly for embedded applications. We build the accelerator for multiple precisions ( FIXED8, FIXED16, FLOAT32 ) of YOLOv3-tiny. The architecture uses a homogenous systolic array architecture with a synchronized pipeline adder tree for convolution, allowing it to be scalable for multiple variants of YOLO with a change in the host driver. It is a deeply pipelined architecture that also exploits three-dimensional spatial parallelism. We evaluated the design on Terasic DE5a-Net-DDR4. The Fixed point (FP-8, FP-16) implementations attain a throughput of 57 GOPs/s (> 23 %) and 46.16 GOPs/s (> 340 %) running at 234 MHz and 227 MHz. We synthesized the first FLOAT32 implementation attaining 11.22 GFLOPs/s, running at 172 MHz

Abstract

Deep Neural Networks (DNNs) have shown remarkable performance in a broad range of computer vision tasks, including in the medical domain. With the advent of DNNs, the medical community has witnessed significant developments in segmentation, classification, and detection. But this success comes with a cost of heavy reliance on the abundance of data. Medical data, however, is often highly limited in volume and quality due to sparsity of patient contact, variability in medical care, and privacy concerns. Hence, to train large networks we seek data from different sources. In such a scenario, it is of interest to design a model that learns continuously and adapts to datasets or tasks as and when they are available. However, one of the important steps to achieve such a never-ending learning process is to overcome Catastrophic Forgetting (CF) of previously seen data or tasks. CF refers to the significant degradation in performance on the old task/dataset. To avoid confusion, we call a training regime Continual Learning (CL) when CAD systems have to handle a sequence of datasets collected over time from different sites with different imaging parameters/populations. Similarly, Incremental Learning (IL) is when CAD systems have to learn new classes as and when new annotations are made available. The work described in this thesis address core aspects of both CL & IL and has been compared against the state-of-the-art methods. In this thesis, we assume that access to the data belonging to previously trained datasets or tasks is not available which makes both CL and IL processes even more challenging. We start with developing a CL system that learns sequentially on different datasets and handles CF using the Uncertainty mechanism. The system consists of an ensemble of models which are trained or finetuned on each dataset and considers the prediction from the model which has the least uncertainty. We then investigate a new way to tackle CF in CL by manifold learning, inspired by the defense mechanisms against adversarial attacks. Our method uses a ‘Reformer’ which is essentially a denoising autoencoder that ‘reforms’ or brings the data from all the datasets together towards a common manifold. These reformed samples are then passed to the network to learn the desired task. Towards IL, we propose a novel approach that ensures that a model remembers the causal factor behind the decisions on the old classes, while incrementally learning new classes. We introduce a common auxiliary task during the course of incremental training, whose hidden representations are shared across all the classification heads. All the experiments for both CL and IL are conducted on multiple datasets and have shown significant performance over the state-of-the-art methods