Abstract

Abstract—This paper presents final results of ICDAR 2019Scene Text Visual Question Answering competition (ST-VQA).ST-VQA introduces an important aspect that is not addressedby any Visual Question Answering system up to date, namelythe incorporation of scene text to answer questions asked aboutan image. The competition introduces a new dataset comprising23,038images annotated with31,791question / answer pairswhere the answer is always grounded on text instances presentin the image. The images are taken from7different publiccomputer vision datasets, covering a wide range of scenarios.The competition was structured in three tasks of increas-ing difficulty, that require reading the text in a scene andunderstanding it in the context of the scene, to correctlyanswer a given question. A novel evaluation metric is presented,which elegantly assesses both key capabilities expected from anoptimal model: text recognition and image understanding.A detailed analysis of results from different participants isshowcased, which provides insight into the current capabilitiesof VQA systems that can read. We firmly believe the datasetproposed in this challenge will be an important milestone toconsider towards a path of more robust and general models thatcan exploit scene text to achieve holistic image understanding.

Abstract

Cross-modal retrieval methods have been significantly improvedin last years with the use of deep neural networks and large-scaleannotated datasets such as ImageNet and Places. However, collect-ing and annotating such datasets requires a tremendous amountof human effort and, besides, their annotations are limited to dis-crete sets of popular visual classes that may not be representativeof the richer semantics found on large-scale cross-modal retrievaldatasets. In this paper, we present a self-supervised cross-modalretrieval framework that leverages as training data the correlationsbetween images and text on the entire set of Wikipedia articles.Our method consists in training a CNN to predict: (1) the semanticcontext of the article in which an image is more probable to appearas an illustration, and (2) the semantic context of its caption. Ourexperiments demonstrate that the proposed method is not only ca-pable of learning discriminative visual representations for solvingvision tasks like classification, but that the learned representationsare better for cross-modal retrieval when compared to supervisedpre-training of the network on the ImageNet dataset.

Abstract

We present a simple, yet effective, Neural Ma-chineTranslationsystemforIndianlanguages.Wedemonstratethefeasibilityformultiplelan-guage pairs, and establish a strong baseline forfurther research.

Abstract

Given a target image as query, person re-identification systems retrieve a ranked list of candidate matches on a per-camera basis. In deployed systems, a human operator scans these lists and labels sighted targets by touch or mouse-based selection. However, classical re-id approaches generate per-camera lists independently. Therefore, target identifications by operator in a subset of cameras cannot be utilized to improve ranking of the target in remaining set of network cameras. To address this shortcoming, we propose a novel sequential multi-camera re-id approach. The proposed approach can accommodate human operator inputs and provides early gains via a monotonic improvement in target ranking. At the heart of our approach is a fusion function which operates on deep feature representations of query and candidate matches. We formulate an optimization procedure custom-designed to incrementally …

Abstract

Abstract—The ICDAR 2019 Challenge on “Scanned receiptsOCR and key information extraction” (SROIE) covers importantaspects related to the automated analysis of scanned receipts.The SROIE tasks play a key role in many document analysissystems and hold significant commercial potential. Although a lotof works have been published over the years on administrativedocument analysis, the community has advanced relatively slowly,as most datasets have been kept private. One of the keycontributions of SROIE to the document analysis community is tooffer a first, standardized dataset of 1000 whole scanned receiptimages and annotations, as well as an evaluation procedure forsuch tasks. The Challenge is structured around three tasks,namely Scanned Receipt Text Localization (Task 1), ScannedReceipt OCR (Task 2) and Key Information Extraction fromScanned Receipts (Task 3). The competition opened on 10thFebruary, 2019 and closed on 5th May, 2019. We received 29,24 and 18 valid submissions received for the three competitiontasks, respectively. This report presents the competition datasets,define the tasks and the evaluation protocols, offer detailedsubmission statistics, as well as an analysis of the submittedperformance. While the tasks of text localization and recognitionseem to be relatively easy to tackle, it is interesting to observethe variety of ideas and approaches proposed for the informationextraction task. According to the submissions’ performance webelieve there is still margin for improving information extractionperformance, although the current dataset would have to growsubstantially in following editions. Given the success of theSROIE competition evidenced by the wide interest generatedand the healthy number of submissions from academic, researchinstitutes and industry over different countries, we consider thatthe SROIE competition can evolve into a useful resource for thecommunity, drawing further attention and promoting researchand development efforts in this field

Abstract

Semantic Segmentation Datasets for ResourceConstrained TrainingAshutosh Mishra1?, Sudhir Kumar1,2?, Tarun Kalluri1,3?, Girish Varma1,Anbumani Subramaian4, Manmohan Chandraker3, and CV Jawahar11IIIT Hyderabad2University at Buffalo, State University of New York3University of California, San Diego4Intel BangaloreAbstract.Several large scale datasets, coupled with advances in deepneural network architectures have been greatly successful in pushingthe boundaries of performance in semantic segmentation in recent years.However, the scale and magnitude of such datasets prohibits ubiquitoususe and widespread adoption of such models, especially in settings withserious hardware and software resource constraints. Through this work,we propose two simple variants of the recently proposed IDD dataset,namelyIDD-miniandIDD-lite, for scene understanding in unstructuredenvironments. Our main objective is to enable research and benchmark-ing in training segmentation models. We believe that this will enablequick prototyping useful in applications like optimum parameter andarchitecture search, and encourage deployment on low resource hardwaresuch as Raspberry Pi. We show qualitatively and quantitatively that withonly 1 hour of training on 4GB GPU memory, we can achieve satisfactorysemantic segmentation performance on the proposed datasets.

Abstract

In recent years, there has been an explosion in the number of memesbeing created and circulated in online social networks. Despite theirrapidly increasing impact on how we communicate online, memeimages are virtually inaccessible to the visually impaired users. Ex-isting automated assistive systems that were primarily devised fornatural photos in social media, overlook the specific fine-grainedvisual details in meme images. In this paper, we concentrate ondescribing one such prominent visual detail: the meme face emo-tion. We propose a novel automated method that enables visuallyimpaired social media users to understand and appreciate memeface emotions with the help of rich textual captions. We first collecta challenging dataset of meme face emotion captions to supportfuture research in face emotion understanding. We design a two-stage approach that significantly outperforms baseline approaches across all the standard captioning metrics and also generates richerdiscriminative captions. By validating our solution with the helpof visually impaired social media users, we show that our emotioncaptions enable them to understand and appreciate one of the mostpopular classes of meme images encountered on the Internet forthe first time. Code, data, and models are publicly available.

Abstract

The popularity of egocentric cameras and their always-on nature has lead to the abundance of day-long first-person videos. Because of the extreme shake and highly redundant nature, these videos are difficult to watch from beginning to end and often require summarization tools for their efficient consumption. However, traditional summarization techniques developed for static surveillance videos, or highly curated sports videos and movies are, either, not suitable or simply do not scale for such hours long videos in the wild. On the other hand, specialized summarization techniques developed for egocentric videos limit their focus to important objects and people. In this paper, we present a novel unsupervised reinforcement learning technique to generate video summaries from day long egocentric videos. Our approach can be adapted to generate summaries of various lengths making it possible to view even 1-minute summaries of one’s entire day. The technique can also be adapted to various rewards, such as distinctiveness and indicativeness of the summary. When using the facial saliency-based reward, we show that our approach generates summaries focusing on social interactions, similar to the current state-of-the-art (SOTA). Quantitative comparison on the benchmark Disney dataset shows that our method achieves significant improvement in Relaxed F-Score (RFS) (32.56 vs. 19.21) and BLEU score (12.12 vs. 10.64). Finally, we show that our technique can be applied for summarizing traditional, short, hand-held videos as well, where we improve the SOTA F-score on benchmark SumMe and TVSum datasets from 41.4 to 45.6 and 57.6 to 59.1 respectively

Abstract

In light of the recent breakthroughs in automatic machine transla-tion systems, we propose a novel approach that we term as "Face-to-Face Translation". As today’s digital communication becomesincreasingly visual, we argue that there is a need for systems thatcan automatically translate a video of a person speaking in lan-guage A into a target language B with realistic lip synchronization.In this work, we create an automatic pipeline for this problem anddemonstrate its impact in multiple real-world applications. First, webuild a working speech-to-speech translation system by bringing together multiple existing modules from speech and language. Wethen move towards "Face-to-Face Translation" by incorporating anovel visual module, LipGAN for generating realistic talking facesfrom the translated audio. Quantitative evaluation of LipGAN onthe standard LRW test set shows that it significantly outperform sexisting approaches across all standard metrics. We also subjectour Face-to-Face Translation pipeline, to multiple human evalua-tions and show that it can significantly improve the overall user experience for consuming and interacting with multimodal content across languages. Code, models and demo video are made publicly available.

Abstract

No Abstract available