Abstract

A Cost Efficient Approach to Correct OCR Errors in Large Document CollectionsDeepayan Das, Jerin Philip, Minesh Mathew and C. V. JawaharCenter for Visual Information Technology, IIIT Hyderabad, India.{deepayan.das, jerin.philip, minesh.mathew}@research.iiit.ac.in, jawahar@iiit.ac.inAbstract—Word error rate of anOCRis often higher thanits character error rate. This is especially true whenOCRsaredesigned by recognizing characters. High word accuracies arecritical for many practical applications like content creationand text-to-speech systems. In order to detect and correct themisrecognised words, it is common for anOCRto employ apost-processor module to improve the word accuracy. However,conventional approaches to post-processing like looking up adictionary or using a statistical language model (SLM), are stilllimited. In many such scenarios, it is often required to removethe outstanding errors manually.We observe that the traditional post-processing schemes lookat error words sequentially sinceOCRs process documents oneat a time. We propose a cost efficient model to address the errorwords in batches rather than correcting them individually.We exploit the fact that a collection of documents (eg. abook), unlike a single document, has a structure leading torepetition of words. Such words, if efficiently grouped togetherand corrected together, can lead to significant reduction in theeffort. Error correction can be fully automatic or with a humanin the loop. We compare the performance of our method withvarious baseline approaches including the case where all theerrors are removed by a human. We demonstrate the efficacy ofour solution empirically by reporting more than70%reductionin the human effort with near perfect error correction. Wevalidate our method on books in both English and Hindi.

Abstract

Palm-leaf manuscripts are one of the oldest medium of inscription in many Asian countries. Especially, manuscripts from the Indian subcontinent form an important part of the world's literary and cultural heritage. Despite their significance, large-scale datasets for layout parsing and targeted annotation systems do not exist. Addressing this, we propose a web-based layout annotation and analytics system. Our system, called HInDoLA, features an intuitive annotation GUI, a graphical analytics dashboard and interfaces with machine-learning based intelligent modules on the backend. HInDoLA has successfully helped us create the first ever large-scale dataset for layout parsing of Indic palm-leaf manuscripts. These manuscripts, in turn, have been used to train and deploy deep-learning based modules for fully automatic and semi-automatic instance-level layout parsing.

Abstract

bstract—Reading of mathematical expression or equationin the document images is very challenging due to the largevariability of mathematical symbols and expressions. In thispaper, we pose reading of mathematical equation as a taskof generation of the textual description which interprets theinternal meaning of this equation. Inspired by the naturalimage captioning problem in computer vision, we presenta mathematical equation description (MED) model, a novelend-to-end trainable deep neural network based approachthat learns to generate a textual description for readingmathematical equation images. OurMEDmodel consists of aconvolution neural network as an encoder that extracts featuresof input mathematical equation images and a recurrent neuralnetwork with attention mechanism which generates descriptionrelated to the input mathematical equation images. Due tothe unavailability of mathematical equation image data setswith their textual descriptions, we generate two data sets forexperimental purpose. To validate the effectiveness of ourMEDmodel, we conduct a real-world experiment to see whether thestudents are able to write equations by only reading or listeningtheir textual descriptions or not. Experiments conclude that thestudents are able to write most of the equations correctly byreading their textual descriptions only.

Abstract

bstract—Graphical elements: particularly tables and fig-ures contain a visual summary of the most valuable infor-mation contained in a document. Therefore, localization ofsuch graphical objects in the document images is the initialstep to understand the content of such graphical objects ordocument images. In this paper, we present a novel end-to-endtrainable deep learning based framework to localize graphicalobjects in the document images called as Graphical ObjectDetection (GOD). Our framework is data-driven and does notrequire any heuristics or meta-data to locate graphical objectsin the document images. TheGODexplores the concept oftransfer learning and domain adaptation to handle scarcity oflabeled training images for graphical object detection task inthe document images. Performance analysis carried out on thevarious public benchmark data sets:ICDAR-2013,ICDAR-POD2017 andUNLVshows that our model yields promising resultsas compared to state-of-the-art techniques

Abstract

Abstract—Automated evaluation of handwritten answers hasbeen a challenging problem for scaling the education systemfor many years. Speeding up the evaluation remains as themajor bottleneck for enhancing the throughput of instructors.This paper describes an effective method for automaticallyevaluating the short descriptive handwritten answers fromthe digitized images. Our goal is to evaluate a student’shandwritten answer by assigning an evaluation score that iscomparable to the human-assigned scores. Existing works inthis domain mainly focused on evaluating handwritten essayswith handcrafted, non-semantic features. Our contribution istwo-fold: 1) we model this problem as a self-supervised, feature-based classification problem, which can fine-tune itself for eachquestion without any explicit supervision. 2) We introduce theusage of semantic analysis for auto-evaluation in handwrittentext space using the combination of Information Retrievaland Extraction (IRE) and, Natural Language Processing (NLP)methods to derive a set of useful features. We tested our methodon three datasets created from various domains, using the helpof students of different age groups. Experiments show that ourmethod performs comparably to that of human evaluators.

Abstract

Given a target image as query, person re-identification systems retrieve a ranked list of candidate matches on a per-camera basis. In deployed systems, a human operator scans these lists and labels sighted targets by touch or mouse-based selection. However, classical re-id approaches generate per-camera lists independently. Therefore, target identifications by operator in a subset of cameras cannot be utilized to improve ranking of the target in remaining set of network cameras. To address this shortcoming, we propose a novel sequential multi-camera re-id approach. The proposed approach can accommodate human operator inputs and provides early gains via a monotonic improvement in target ranking. At the heart of our approach is a fusion function which operates on deep feature representations of query and candidate matches. We formulate an optimization procedure custom-designed to incrementally …

Abstract

Abstract—The ICDAR 2019 Challenge on “Scanned receiptsOCR and key information extraction” (SROIE) covers importantaspects related to the automated analysis of scanned receipts.The SROIE tasks play a key role in many document analysissystems and hold significant commercial potential. Although a lotof works have been published over the years on administrativedocument analysis, the community has advanced relatively slowly,as most datasets have been kept private. One of the keycontributions of SROIE to the document analysis community is tooffer a first, standardized dataset of 1000 whole scanned receiptimages and annotations, as well as an evaluation procedure forsuch tasks. The Challenge is structured around three tasks,namely Scanned Receipt Text Localization (Task 1), ScannedReceipt OCR (Task 2) and Key Information Extraction fromScanned Receipts (Task 3). The competition opened on 10thFebruary, 2019 and closed on 5th May, 2019. We received 29,24 and 18 valid submissions received for the three competitiontasks, respectively. This report presents the competition datasets,define the tasks and the evaluation protocols, offer detailedsubmission statistics, as well as an analysis of the submittedperformance. While the tasks of text localization and recognitionseem to be relatively easy to tackle, it is interesting to observethe variety of ideas and approaches proposed for the informationextraction task. According to the submissions’ performance webelieve there is still margin for improving information extractionperformance, although the current dataset would have to growsubstantially in following editions. Given the success of theSROIE competition evidenced by the wide interest generatedand the healthy number of submissions from academic, researchinstitutes and industry over different countries, we consider thatthe SROIE competition can evolve into a useful resource for thecommunity, drawing further attention and promoting researchand development efforts in this field

Abstract

Abstract—Document figure classification (DFC) is an im-portant stage of a document figure understanding system.The design of a DFC system required a well defined figurecategories and dataset. To the best of the author’s knowledge,the existing datasets related to classification of figures in thedocument images are limited with respect to their size andcategories [1]–[3]. In this paper, we introduce a scientificfigure classification dataset, named asDocFigure. The datasetconsists of33K annotated figures of28different categoriespresent in the document images which correspond to scientificarticles published in theCVPR,ECCV,ICCV, etc. conferences inlast several years. Manual annotation of such a large number(33K) of figures is time consuming and cost ineffective. Inthis article, we design a web based annotation tool which canefficiently assign category labels to large number of figures withthe minimum efforts of human annotators. To benchmark ourgenerated dataset on classification task, we propose three base-line classification techniques using deep feature, deep texturefeature and combination of both. In our analysis, we found thatthe combination of both deep feature and deep texture featureis more effective for document figure classification task thanthe individual features. The dataset and the code are publiclyavailable at https://researchweb.iiit.ac.in/∼jobin.kv/projects

Abstract

Semantic Segmentation Datasets for ResourceConstrained TrainingAshutosh Mishra1?, Sudhir Kumar1,2?, Tarun Kalluri1,3?, Girish Varma1,Anbumani Subramaian4, Manmohan Chandraker3, and CV Jawahar11IIIT Hyderabad2University at Buffalo, State University of New York3University of California, San Diego4Intel BangaloreAbstract.Several large scale datasets, coupled with advances in deepneural network architectures have been greatly successful in pushingthe boundaries of performance in semantic segmentation in recent years.However, the scale and magnitude of such datasets prohibits ubiquitoususe and widespread adoption of such models, especially in settings withserious hardware and software resource constraints. Through this work,we propose two simple variants of the recently proposed IDD dataset,namelyIDD-miniandIDD-lite, for scene understanding in unstructuredenvironments. Our main objective is to enable research and benchmark-ing in training segmentation models. We believe that this will enablequick prototyping useful in applications like optimum parameter andarchitecture search, and encourage deployment on low resource hardwaresuch as Raspberry Pi. We show qualitatively and quantitatively that withonly 1 hour of training on 4GB GPU memory, we can achieve satisfactorysemantic segmentation performance on the proposed datasets.

Abstract

In recent years, there has been an explosion in the number of memesbeing created and circulated in online social networks. Despite theirrapidly increasing impact on how we communicate online, memeimages are virtually inaccessible to the visually impaired users. Ex-isting automated assistive systems that were primarily devised fornatural photos in social media, overlook the specific fine-grainedvisual details in meme images. In this paper, we concentrate ondescribing one such prominent visual detail: the meme face emo-tion. We propose a novel automated method that enables visuallyimpaired social media users to understand and appreciate memeface emotions with the help of rich textual captions. We first collecta challenging dataset of meme face emotion captions to supportfuture research in face emotion understanding. We design a two-stage approach that significantly outperforms baseline approaches across all the standard captioning metrics and also generates richerdiscriminative captions. By validating our solution with the helpof visually impaired social media users, we show that our emotioncaptions enable them to understand and appreciate one of the mostpopular classes of meme images encountered on the Internet forthe first time. Code, data, and models are publicly available.