Abstract

We present a framework for learning an efficient holistic representation for handwritten word images. The proposed method uses a deep convolutional neural network with traditional classification loss. The major strengths of our work lie in: (i) the efficient usage of synthetic data to pre-train a deep network, (ii) an adapted version of the ResNet-34 architecture with the region of interest pooling (referred to as HWNet v2) which learns discriminative features for variable sized word images, and (iii) a realistic augmentation of training data with multiple scales and distortions which mimics the natural process of handwriting. We further investigate the process of transfer learning to reduce the domain gap between synthetic and real domain, and also analyze the invariances learned at different layers of the network using visualization techniques proposed in the literature.

Abstract

For an autonomous corridor following task where the environment is continuously changing, several forms of environmental noise prevent an automated feature extraction procedure from performing reliably. Moreover, in cases where pre-defined features are absent from the captured data, a well defined control signal for performing the servoing task fails to get produced. In order to overcome these drawbacks, we present in this work, using a convolutional neural network (CNN) to directly estimate the required control signal from an image, encompassing feature extraction and control law computation into one single end-to-end framework. In particular, we study the task of autonomous corridor following using a CNN and present clear advantages in cases where a traditional method used for performing the same task fails to give a reliable outcome. We evaluate the performance of our method on this task on a Wheelchair Platform developed at our institute for this purpose.

Abstract

While several datasets for autonomous navigation have become available in recent years, they have tended to focus on structured driving environments. This usually corresponds to well-delineated infrastructure such as lanes, a small number of well-defined categories for traffic participants, low variation in object or background appearance and strong adherence to traffic rules. We propose DS, a novel dataset for road scene understanding in unstructured environments where the above assumptions are largely not satisfied. It consists of 10,004 images, finely annotated with 34 classes collected from 182 drive sequences on Indian roads. The label set is expanded in comparison to popular benchmarks such as Cityscapes, to account for new classes. It also reflects label distributions of road scenes significantly different from existing datasets, with most classes displaying greater within-class diversity. Consistent with …

Abstract

No Abstract available

Abstract

In light of the recent breakthroughs in automatic machine transla-tion systems, we propose a novel approach that we term as "Face-to-Face Translation". As today’s digital communication becomesincreasingly visual, we argue that there is a need for systems thatcan automatically translate a video of a person speaking in lan-guage A into a target language B with realistic lip synchronization.In this work, we create an automatic pipeline for this problem anddemonstrate its impact in multiple real-world applications. First, webuild a working speech-to-speech translation system by bringing together multiple existing modules from speech and language. Wethen move towards "Face-to-Face Translation" by incorporating anovel visual module, LipGAN for generating realistic talking facesfrom the translated audio. Quantitative evaluation of LipGAN onthe standard LRW test set shows that it significantly outperform sexisting approaches across all standard metrics. We also subjectour Face-to-Face Translation pipeline, to multiple human evalua-tions and show that it can significantly improve the overall user experience for consuming and interacting with multimodal content across languages. Code, models and demo video are made publicly available.

Abstract

The popularity of egocentric cameras and their always-on nature has lead to the abundance of day-long first-person videos. Because of the extreme shake and highly redundant nature, these videos are difficult to watch from beginning to end and often require summarization tools for their efficient consumption. However, traditional summarization techniques developed for static surveillance videos, or highly curated sports videos and movies are, either, not suitable or simply do not scale for such hours long videos in the wild. On the other hand, specialized summarization techniques developed for egocentric videos limit their focus to important objects and people. In this paper, we present a novel unsupervised reinforcement learning technique to generate video summaries from day long egocentric videos. Our approach can be adapted to generate summaries of various lengths making it possible to view even 1-minute summaries of one’s entire day. The technique can also be adapted to various rewards, such as distinctiveness and indicativeness of the summary. When using the facial saliency-based reward, we show that our approach generates summaries focusing on social interactions, similar to the current state-of-the-art (SOTA). Quantitative comparison on the benchmark Disney dataset shows that our method achieves significant improvement in Relaxed F-Score (RFS) (32.56 vs. 19.21) and BLEU score (12.12 vs. 10.64). Finally, we show that our technique can be applied for summarizing traditional, short, hand-held videos as well, where we improve the SOTA F-score on benchmark SumMe and TVSum datasets from 41.4 to 45.6 and 57.6 to 59.1 respectively

Abstract

In recent years, there has been an explosion in the number of memesbeing created and circulated in online social networks. Despite theirrapidly increasing impact on how we communicate online, memeimages are virtually inaccessible to the visually impaired users. Ex-isting automated assistive systems that were primarily devised fornatural photos in social media, overlook the specific fine-grainedvisual details in meme images. In this paper, we concentrate ondescribing one such prominent visual detail: the meme face emo-tion. We propose a novel automated method that enables visuallyimpaired social media users to understand and appreciate memeface emotions with the help of rich textual captions. We first collecta challenging dataset of meme face emotion captions to supportfuture research in face emotion understanding. We design a two-stage approach that significantly outperforms baseline approaches across all the standard captioning metrics and also generates richerdiscriminative captions. By validating our solution with the helpof visually impaired social media users, we show that our emotioncaptions enable them to understand and appreciate one of the mostpopular classes of meme images encountered on the Internet forthe first time. Code, data, and models are publicly available.

Abstract

Semantic Segmentation Datasets for ResourceConstrained TrainingAshutosh Mishra1?, Sudhir Kumar1,2?, Tarun Kalluri1,3?, Girish Varma1,Anbumani Subramaian4, Manmohan Chandraker3, and CV Jawahar11IIIT Hyderabad2University at Buffalo, State University of New York3University of California, San Diego4Intel BangaloreAbstract.Several large scale datasets, coupled with advances in deepneural network architectures have been greatly successful in pushingthe boundaries of performance in semantic segmentation in recent years.However, the scale and magnitude of such datasets prohibits ubiquitoususe and widespread adoption of such models, especially in settings withserious hardware and software resource constraints. Through this work,we propose two simple variants of the recently proposed IDD dataset,namelyIDD-miniandIDD-lite, for scene understanding in unstructuredenvironments. Our main objective is to enable research and benchmark-ing in training segmentation models. We believe that this will enablequick prototyping useful in applications like optimum parameter andarchitecture search, and encourage deployment on low resource hardwaresuch as Raspberry Pi. We show qualitatively and quantitatively that withonly 1 hour of training on 4GB GPU memory, we can achieve satisfactorysemantic segmentation performance on the proposed datasets.

Abstract

Abstract—Document figure classification (DFC) is an im-portant stage of a document figure understanding system.The design of a DFC system required a well defined figurecategories and dataset. To the best of the author’s knowledge,the existing datasets related to classification of figures in thedocument images are limited with respect to their size andcategories [1]–[3]. In this paper, we introduce a scientificfigure classification dataset, named asDocFigure. The datasetconsists of33K annotated figures of28different categoriespresent in the document images which correspond to scientificarticles published in theCVPR,ECCV,ICCV, etc. conferences inlast several years. Manual annotation of such a large number(33K) of figures is time consuming and cost ineffective. Inthis article, we design a web based annotation tool which canefficiently assign category labels to large number of figures withthe minimum efforts of human annotators. To benchmark ourgenerated dataset on classification task, we propose three base-line classification techniques using deep feature, deep texturefeature and combination of both. In our analysis, we found thatthe combination of both deep feature and deep texture featureis more effective for document figure classification task thanthe individual features. The dataset and the code are publiclyavailable at https://researchweb.iiit.ac.in/∼jobin.kv/projects

Abstract

Abstract—The ICDAR 2019 Challenge on “Scanned receiptsOCR and key information extraction” (SROIE) covers importantaspects related to the automated analysis of scanned receipts.The SROIE tasks play a key role in many document analysissystems and hold significant commercial potential. Although a lotof works have been published over the years on administrativedocument analysis, the community has advanced relatively slowly,as most datasets have been kept private. One of the keycontributions of SROIE to the document analysis community is tooffer a first, standardized dataset of 1000 whole scanned receiptimages and annotations, as well as an evaluation procedure forsuch tasks. The Challenge is structured around three tasks,namely Scanned Receipt Text Localization (Task 1), ScannedReceipt OCR (Task 2) and Key Information Extraction fromScanned Receipts (Task 3). The competition opened on 10thFebruary, 2019 and closed on 5th May, 2019. We received 29,24 and 18 valid submissions received for the three competitiontasks, respectively. This report presents the competition datasets,define the tasks and the evaluation protocols, offer detailedsubmission statistics, as well as an analysis of the submittedperformance. While the tasks of text localization and recognitionseem to be relatively easy to tackle, it is interesting to observethe variety of ideas and approaches proposed for the informationextraction task. According to the submissions’ performance webelieve there is still margin for improving information extractionperformance, although the current dataset would have to growsubstantially in following editions. Given the success of theSROIE competition evidenced by the wide interest generatedand the healthy number of submissions from academic, researchinstitutes and industry over different countries, we consider thatthe SROIE competition can evolve into a useful resource for thecommunity, drawing further attention and promoting researchand development efforts in this field