Abstract

In this paper, we present AUTORATE, a system that leverages the front camera of a windshield-mounted smartphone to monitor driver’s attention by combining several features. We derive a driver attention rating by fusing spatio-temporal features based on the driver state and behavior such as head pose, eye gaze, eye closure, yawns, use of cellphones, etc. We perform extensive evaluation of AUTORATE on real-world driving data and also data from controlled, static vehicle settings with 30 drivers in a large city. We compare AUTORATE’s automatically-generated rating with the scores given by 5 human annotators. We compute the agreement between AUTORATE’s rating and human annotator rating using kappa coefficient. AUTORATE’s automatically-generated rating has an overall agreement of 0.88 with the ratings provided by 5 human annotators. We also propose soft attention mechanism in AUTORATE which improves AUTORATE’s accuracy by 10%. We use temporal and spatial attention to visualize the key frame and the key action which justify the model’s predicted rating. Further, we observe that personalization in AUTORATE can improve driver specific results by a significant amount

Abstract

We consider a setting where public resources are to be allocated among competing and strategic agents so as to maximize social welfare (the objects should be allocated to those who value them the most). This is called allocative efficiency (AE). We need the agents to report their valuations for obtaining these resources, truthfully referred to as dominant strategy incentive compatibility (DSIC). We use auction-based mechanisms to achieve AE and DSIC yet budget balance cannot be ensured, due to Green-Laffont Impossibility Theorem. That is, the net transfer of money cannot be zero. This problem has been addressed by designing a redistribution mechanism so as to ensure a minimum surplus of money as well as AE and DSIC. The objective could be to minimize surplus in expectation or in the worst case and these objects could be homogeneous or heterogeneous. Designing redistribution mechanisms which perform well in expectation becomes analytically challenging for heterogeneous settings. In this paper, we take a completely different, data-driven approach. We train a neural network to determine an optimal redistribution mechanism based on given settings with both the objectives, optimal in expectation and optimal in the worst case. We also propose a loss function to train a neural network to optimize worst case. We design neural networks with the underlying rebate functions being linear as well as nonlinear in terms of bids of the agents. Our networks' performances are same as the theoretical guarantees for the cases where it has been solved. We observe that a neural network based redistribution mechanism for homogeneous …

Abstract

Deep convolutional features for word images and textual embedding schemes have shown great success in word spotting. In this work, we follow these motivations to propose an End2End embedding framework which jointly learns both the text and image embeddings using state of the art deep convolutional architectures. The three major contributions of this work are: (i) an End2End embedding scheme to learn a common representation for word images and its labels, (ii) building a state of art word image descriptor and demonstrating its utility as off-the-shelf features for word spotting, and (iii) use of synthetic data as a complementary modality to further enhance word spotting and recognition. On the challenging IAM handwritten dataset, we report a mAP of 0.9509 for query-by-string based retrieval task. Under lexicon based word recognition, our proposed method report a 2.66 and 5.10 CER and WER respectively.

Abstract

This paper explores the effectiveness of deep features for doc- ument image segmentation. The document image segmentation problem is modelled as a pixel labeling task where each pixel in the document im- age is classified into one of the predefined labels such as text, comments, decorations and background. Our method first extracts deep features from superpixels of the document image. Then we learn an svm classifier using these features, and segment the document image. Fisher vector encoded convolutional layer features (fv-cnn) and fully connected layer features (fc-cnn) are used in our study. Experiments validate that our method is effective and yields better results for segmenting document images in comparison to the popular approaches on benchmark hand- written datasets.

Abstract

Handwriting recognition (HWR) in Indicscripts, like Devanagari is very challenging due to thesubtleties in the scripts, variations in rendering and the cur-sive nature of the handwriting. Lack of public handwritingdatasets in Indic scripts has long stymied the development ofoffline handwritten word recognizers and made comparisonacross different methods a tedious task in the field. Inthis paper, we release a new handwritten word dataset forDevanagari, IIIT-HW-Dev to alleviate some of these issues.We benchmark the IIIT-HW-Dev dataset using a CNN-RNN hybrid architecture. Furthermore, using this architec-ture, we empirically show that usage of synthetic data andcross lingual transfer learning helps alleviate the issue of lackof training data. We use this proposed pipeline on a publicdataset, RoyDB and achieve state of the art results.

Abstract

Discrete labeling problems are often solvedby formulating them as an integer program,and relaxing the integrality constraint to acontinuous domain. While the continuous re-laxation is closely related to the original in-teger program, its optimal solution is oftenfractional. Thus, the success of a relaxationdepends crucially on the availability of an ac-curate rounding procedure. The problem ofidentifying an accurate rounding procedurehas mainly been tackled in the theoreticalcomputer science community through math-ematical analysis of the worst-case. However,this approach is both onerous and ignores thedistribution of the data encountered in prac-tice. We present a novel interpretation ofrounding procedures as sampling from a la-tent variable model, which opens the door tothe use of powerful machine learning formu-lations in their design. Inspired by the recentsuccess of deep latent variable models we pa-rameterize rounding procedures as a neuralnetwork, which lends itself to efficient opti-mization via back-propagation. By minimiz-ing the expected value of the objective of thediscrete labeling problem over training sam-ples, we learn a rounding procedure that ismore suited to the task at hand. Using bothsynthetic and real world data sets, we demon-strate that our approach can outperform thestate-of-the-art hand-designed rounding pro-cedures

Abstract

Our goal is to spot words in silent speech videos withoutexplicitly recognizing the spoken words, where the lip mo-tion of the speaker is clearly visible and audio is absent. Ex-isting work in this domain has mainly focused on recogniz-ing a fixed set of words in word-segmented lip videos, whichlimits the applicability of the learned model due to limitedvocabulary and high dependency on the model’s recogni-tion performance.Our contribution is two-fold: 1) we develop a pipelinefor recognition-free retrieval, and show its performanceagainst recognition-based retrieval on a large-scale datasetand another set of out-of-vocabulary words. 2) We intro-duce a query expansion technique using pseudo-relevantfeedback and propose a novel re-ranking method based onmaximizing the correlation between spatio-temporal land-marks of the query and the top retrieval candidates. Ourword spotting method achieves 35% higher mean aver-age precision over recognition-based method on large-scaleLRW dataset. Finally, we demonstrate the application of themethod by word spotting in a popular speech video (“Thegreat dictator” by Charlie Chaplin) where we show that theword retrieval can be used to understand what was spokenperhaps in the silent movies

Abstract

The accuracy of information retrieval systems is often measured using complex loss functions such as the average precision (AP) or the normalized discounted cumulative gain (NDCG). Given a set of positive and negative samples, the parameters of a retrieval system can be estimated by minimizing these loss functions. However, the non-differentiability and non-decomposability of these loss functions does not allow for simple gradient based optimization algorithms. This issue is generally circumvented by either optimizing a structured hinge-loss upper bound to the loss function or by using asymptotic methods like the direct-loss minimization framework. Yet, the high computational complexity of loss-augmented inference, which is necessary for both the frameworks, prohibits its use in large training data sets. To alleviate this deficiency, we present a novel quicksort flavored algorithm for a large class of non-decomposable loss functions. We provide a complete characterization of the loss functions that are amenable to our algorithm, and show that it includes both AP and NDCG based loss functions. Furthermore, we prove that no comparison based algorithm can improve upon the computational complexity of our approach asymptotically. We demonstrate the effectiveness of our approach in the context of optimizing the structured hinge loss upper bound of AP and NDCG loss for learning models for a variety of vision tasks. We show that our approach provides significantly better results than simpler decomposable loss functions, while requiring a comparable training time.

Abstract

With the increase in the number of students enrolled in the university system, regular assessment of student performance has become challenging. This is specially true in case of summative assessments, where one expects the student to write down an answer on paper, rather than selecting a correct answer from multiple choices. In this paper, we present a document image workflow system that helps in scaling the handwritten student assessments in a typical university setting. We argue that this improves the efficiency since the book keeping time as well as physical paper movement is minimized. An electronic workflow can make the anonymization easy, alleviating the fear of biases in many cases. Also, parallel and distributed assessment by multiple instructors is straightforward in an electronic workflow system. At the heart of our solution, we have (i) a distributed image capture module with a mobile phone (ii) image processing algorithms that improve the quality and readability (iii) image annotation module that process the evaluations/feedbacks as a separate layer. Our system also acts as a platform for modern image analysis which can be adapted to the domain of student assessments. This include (i) Handwriting recognition and word spotting [5] (ii) Measure of document similarity [6] (iii) Aesthetic analysis of handwriting [7] (iv) Identity of the writer [4] etc. With the handwriting assessment workflow system, all these recent advances in computer vision can become practical and applicable in evaluating student assessments

Abstract

Deep CNNs for semantic segmentation have high memory and run time requirements. Various approaches have been proposed to make CNNs efficient like grouped, shuffled, depth-wise separable convolutions. We study the effectiveness of these techniques on a real-time semantic segmentation architecture like ERFNet for improving runtime by over 5X. We apply these techniques to CNN layers partially or fully and evaluate the testing accuracies on Cityscapes dataset. We obtain accuracy vs parameters/FLOPs trade offs, giving accuracy scores for models that can run under specified runtime budgets. We further propose a novel training procedure which starts out with a dense convolution but gradually evolves towards a grouped convolution. We show that our proposed training method and efficient architecture design can improve accuracies by over 8% with depthwise separable convolutions applied on the encoder of ERFNet and attaching a light weight decoder. This results in a model which has a 5X improvement in FLOPs while only suffering a 4% degradation in accuracy with respect to ERFNet.