Abstract

Understanding the nature of data is the key to build-ing good representations. In domains such as natural im-ages, the data comes from very complex distributions which are hard to capture. Feature learning intends to discover or best approximate these underlying distributions and use their knowledge to weed out irrelevant information, pre-serving most of the relevant information. Feature learning can thus be seen as a form of dimensionality reduction. In this paper, we describe a feature learning scheme for nat-ural images. We hypothesize that image patches do not all come from the same distribution, they lie in multiple non-linear subspaces. We propose a framework that uses KRestricted Boltzmann Machines (K-RBMS) to learn mul-tiple non-linear subspaces in the raw image space. Pro-jections of the image patches into these subspaces gives us features, which we use to build image representations.Our algorithm solves the coupled problem of finding the right non-linear subspaces in the input space and associ-ating image patches with those subspaces in an iterative EM like algorithm to minimize the overall reconstruction error. Extensive empirical results over several popular im-age classification datasets show that representations based on our framework outperform the traditional feature repre-sentations such as the SIFT based Bag-of-Words (BoW) and convolutional deep belief networks.

Abstract

The automatic discovery of distinctive parts for an ob-ject or scene class is challenging since it requires simulta-neously to learn the part appearance and also to identify the part occurrences in images. In this paper, we propose a simple, efficient, and effective method to do so. We ad-dress this problem by learning parts incrementally, starting from a single part occurrence with an Exemplar SVM. In this manner, additional part instances are discovered and aligned reliably before being considered as training exam-ples. We also propose entropy-rank curves as a means of evaluating the distinctiveness of parts shareable between categories and use them to select useful parts out of a set of candidates.We apply the new representation to the task of scene cat-egorisation on the MIT Scene 67 benchmark. We show that our method can learn parts which are significantly more in-formative and for a fraction of the cost, compared to previ-ous part-learning methods such as Singhet al. [28]. We also show that a well constructed bag of words or Fisher vector model can substantially outperform the previous state-of-the-art classification performance on this data.

Abstract

Automatically generating meaningful descriptions for images has recently emerged as an important area of re-search. In this direction, a nearest-neighbour based gener-ative phrase prediction model (PPM) proposed by (Guptaet al. 2012) was shown to achieve state-of-the-art results on PASCAL sentence dataset, thanks to the simultaneous use of three different sources of information (i.e. visual clues,corpus statistics and available descriptions). However, they do not utilize semantic similarities among the phrases that might be helpful in relating semantically similar phrases during phrase relevance prediction. In this paper, we ex-tend their model by considering inter-phrase semantic sim-ilarities. To compute similarity between two phrases, we consider similarities among their constituent words deter-mined using Word Net. We also re-formulate their objective function for parameter learning by penalizing each pair of phrases unevenly, in a manner similar to that in structured predictions. Various automatic and human evaluations are performed to demonstrate the advantage of our “semantic phrase prediction model” (SPPM) over PPM.

Abstract

We focus on the problem of mining object categories from large datasets like Google Street View images. Mining ob-ject categories in these unannotated datasets is an impor-tant and useful step to extract meaningful information. Of-ten the location and spatial extent of an object in an im-age is unknown. Mining objects in such a setting is hard.Recent methods model this problem as learning a separate classifier for each category. This is computationally expen-sive since a large number of classifiers are required to be trained and evaluated, before one can mine a concise set of meaningful objects. On the other hand, fast and efficient solutions have been proposed for the retrieval of instances(same object) from large databases. We borrow, from in-stance retrieval pipeline, its strengths and adapt it to speedup category mining. For this, we explore objects which are“near-instances”.We mine several near-instance object categories from images. Using an instance retrieval based solution, we are able to mine certain categories of near-instance objects much faster than an Exemplar SVM based solution.

Abstract

We aim to decompose a global histogram representation of an image into histograms of its associated objects and re-gions. This task is formulated as an optimization problem,given a set of linear classifiers, which can effectively dis-criminate the object categories present in the image. Our decomposition bypasses harder problems associated with accurately localizing and segmenting objects. We evaluate our method on a wide variety of composite histograms, and also compare it with MRF-based solutions. In addition to merely mea suring the accuracy of decomposition, we also show the utility of the estimated object and background his-tograms for the task of image classification on thePASCALVOC2007dataset.

Abstract

We present an approach for the text-to-image retrieval problem based on textual content present in images. Given the recent developments in understanding text in images, an appealing approach to address this problem is to localize and recognize the text, and then query the database, as in a text retrieval problem. We show that such an approach, de-spite being based on state-of-the-art methods, is insufficient,and propose a method, where we do not rely on an exact lo-calization and recognition pipeline. We take a query-driven search approach, where we find approximate locations of characters in the text query, and then impose spatial con-straints to generate a ranked list of images in the database.The retrieval performance is evaluated on public scene text datasets as well as three large datasets, namely IIIT scene text retrieval, Sports-10K and TV series-1M, we introduce.

Abstract

Existing mobile image instance retrieval applications as-sume a network-based usage where image features are sent to a server to query an online visual database. In this scenario, there are no restrictions on the size of the vi-sual database. This paper, however, examines how to per-form this same task offline, where the entire visual index must reside on the mobile device itself within a small mem-ory footprint. Such solutions have applications on location recognition and product recognition. Mobile instance re-trieval requires a significant reduction in the visual index size. To achieve this, we describe a set of strategies that can reduce the visual index up to 60-80×compared to a standard instance retrieval implementation found on desk-tops or servers. While our proposed reduction steps affect the overall mean Average Precision (mAP), they are ableto maintain a good Precision for the top Kresults (PK).We argue that for such offline application, maintaining agood PKis sufficient. The effectiveness of this approach is demonstrated on several standard databases. A working application designed for a remote historical site is also pre-sented. This application is able to reduce an 50,000 image index structure to 25 MBs while providing a precision of97%forP10and 100%forP1

Abstract

Error detection in OCR output using dictionaries and statistical language models (SLMs) have become common practice for some time now, while designing post-processors. Mul-tiple strategies have been used successfully in English to achieve this. However, this has not yet translated towards improving error detection performance in many inflectional languages, specially Indian languages. Challenges such as large unique word list, lack of linguistic resources, lack of reliable language models, etc. are some of the reasons for this. In this paper, we investigate the ma-jor challenges in developing error detection techniques for highly inflectional Indian languages. We compare and contrast several attributes of English with inflectional languages such as Telugu and Malayalam. We make observations by analyzing statistics computed from popular corpora and relate these observations to the error detection schemes. We propose a method which can detect errors for Telugu and Malayalam, with an F-Score comparable to some of the less inflectional languages like Hindi.Our method learns from the error patterns and SLMs.

Abstract

n this paper, we present a solution towards building a retrieval system over handwritten document images that i) is recognition-free, ii) allows text-querying, iii) can retrieve at sub-word level, iv) can search for out-of-vocabulary words. Unlike previous approaches that operate at either character or word lev-els, we use character n-gram images (CNG-img) as the retrieval primitive.CNG-img are sequences of character segments, that are represented and matched in the image-space. The word-images are now treated as a bag-of-CNG-img, that can be indexed and matched in the feature space. This allows for recognition-free search (query-by-example), which can retrieve morphologically similar words that have matching sub-words. Further, to enable query-by-keyword, we build an automated scheme to generate labeled exemplars for characters and character n-grams, fromun constrained handwritten documents. We pose this problem as one of weakly-supervised learning, where character/n-gram labeling is obtained automatically from the word labels. There sulting retrieval system can answer queries from an unlimited vocabulary. The approach is demonstrated on the George Wash-ington collection, results show major improvement in retrieval performance as compared to word-recognition and word-spotting methods

Abstract

Optical Character Recognition (OCR) problems areoften formulated as isolated character (symbol) classification task followed by a post-classification stage (which contains modules like Unicode generation, error correction etc. ) to generate the textual representation, for most of the Indian scripts. Such approaches are prone to failures due to (i) difficulties in designing reliable word-to-symbol segmentation module that can robustly work in presence of degraded (cut/fused) images and (ii) convert-ing the outputs of the classifiers to a valid sequence of Unicodes.In this paper, we propose a formulation, where the expectations on these two modules is minimized, and the harder recognition task is modelled as learning of an appropriate sequence to sequence translation scheme. We thus formulate the recognition as a direct transcription problem. Given many examples of feature sequences and their corresponding Unicode representations, our objective is to learn a mapping which can convert a word directly in to a Unicode sequence. This formulation has multiple practical advantages: (i) This reduces the number of classes significantly for the Indian scripts. (ii) It removes the need for a reliable word-to-symbol segmentation. (ii) It does not require strong annotation of symbols to design the classifiers, and (iii) It directly generates a valid sequence of Unicodes. We test our method on more than 6000 pages of printed Devanagari documents from multiple sources. Our method consistently outperforms other state of the art implementations.