Abstract

A processor-implemented method for generating a lip-sync for a face to a target speech of a live session to a speech in one or more languages in-sync with improved visual quality using a machine learning model and a pre-trained lip-sync model is provided. The method includes (i) determining a visual representation of the face and an audio representation, the visual representation includes crops of the face; (ii) modifying the crops of the face to obtain masked crops; (iii) obtaining a reference frame from the visual representation at a second timestamp; (iv) combining the masked crops at the first timestamp with the reference to obtain lower half crops; (v) training the machine learning model by providing historical lower half crops and historical audio representations as training data; (vi) generating lip-synced frames for the face to the target speech, and (vii) generating an in-sync lip-synced frames by the pre-trained lip-sync model.

Abstract

Transfer learning is a popular method for obtaining deep trained models for data-scarce face tasks such as head pose and emotion. However, current transfer learning methods are inefficient and time-consuming as they do not fully account for the relationships between related tasks. Moreover, the transferred model is large and computationally expensive. As an alternative, we propose ETL: a technique that efficiently transfers a pre-trained model to a new task by retaining only cross-task aware filters, resulting in a sparse transferred model. We demonstrate the effectiveness of ETL by transferring VGGFace, a popular face recognition model to four diverse face tasks. Our experiments show that we attain a size reduction up to 97% and an inference time reduction up to 94% while retaining 99.5% of the baseline transfer learning accuracy

Abstract

Deep learning has shown remarkable performances in image classification, including those of plants and leaves. However, high-performing networks in terms of accuracy may not be using the salient regions for making the prediction and could be prone to biases. This work proposes a neural network architecture incorporating handcrafted features and fusing them with the learned features. Using hybrid features provides better control and understanding of the feature space while leveraging deep learning capabilities. Furthermore, a new IoU-based metric is introduced to assess the CNN-based classifier’s performance based on the regions focused on making predictions. Experiments over multiple leaf disease datasets demonstrate the performance improvement with the model using hybrid features. Classification using hybrid features performed better than the baseline models in terms of P@1 and also on the IoU …

Abstract

Note-taking methods and devices have improved tremendously over the past few decades, and people are finding new ways to write notes and take photos. Automatic extraction, recognition, and retrieval are necessary to process the huge chunk of digitized document data. However, an important step in all of these pipelines is the pre-processing step, mainly image enhancement or clean-up, which enhances the text regions and suppresses the non-text regions. In this article, we look at the problem of image enhancement or clean-up on one such important class of images (i.e., mobile captured glass board images). We present a simple yet efficient algorithm using the concepts of classical image processing techniques to solve the problem, and the obtained results are promising in comparison to the Office Lens.

Abstract

Binarization is an essential pre-processing step for many document image analysis tasks. Binarization of handwritten documents is more challenging than printed documents because of the non-uniform density of ink and the variable thickness of strokes. Instead of traditional scanners, people nowadays use the mobile camera to capture documents, including text written on white and glass boards. The quality of the camera-captured document images is often poor when compared with scanned document images. This impacts binarization accuracy. This paper presents a deep learning-based binarization framework called Deep Semantic Binarization (DSB) to binarize various document images. We pose document image binarization problem as a pixel-wise two-class classification task. Deep networks (including DSB) require many training images during training. However, the benchmark datasets with a limited number of training images are publicly available in the literature. We explore various training strategies, including transfer learning, to handle the data scarcity during training. Due to the unavailability of mobile-captured whiteboard and glass board images, we created two datasets, namely WBIDS-IIIT and GBIDS-IIIT, with associated ground truths. We validate DSBN on the public benchmark DIBCO dataset and WBIDS-IIIT and GBIDS-IIIT datasets. We empirically demonstrate that the DSB outperforms the state-of-the-art techniques for WBIDS-IIIT, GBIDS-IIIT and public datasets.

Abstract

Recent development in deep learning-based recognizers needs a large annotated corpus for creating the model. Manually annotating a large corpus is time-consuming, costly, and tedious. In this work, we propose a framework for automatic annotation at the word level for given handwritten data and corresponding text sequences (or corpora). The proposed framework consists of five modules (i) pre-processing,(ii) word detection,(iii) word recognition,(iv) alignment, and (v) manual correction and verification. The preprocessing module cleans the image and crops the text region from an image. Word detection and recognition modules localize and recognize words. It is necessary to align words in the sequence with the word images during detection and recognition because of errors in writing. The alignment module aligns words in text sequence to the word images. The human annotator will correct the errors in the automatic annotation process and verify the document. Finally, we created an annotated dataset containing word images and their corresponding ground truth transcriptions. In this work, we demonstrate the proposed tool for annotating 14 sets corresponding to 13 Indic languages and English. Each set contains 15000 handwritten document images. On an extensive collection of handwritten document images in 14 languages, 80% of words are correctly annotated by the automatic annotation tool, while the remaining 20% are corrected manually.

Abstract

Recognition of text on word or line images, without the need for sub-word segmentation has become the mainstream of research and development of text recognition for Indian languages. Modelling unsegmented sequences using Connectionist Temporal Classification (CTC) is the most commonly used approach for segmentation-free OCR. In this work we present a comprehensive empirical study of various neural network models that uses CTC for transcribing step-wise predictions in the neural network output to a Unicode sequence. The study is conducted for 13 Indian languages, using an internal dataset that has around 1000 pages per language. We study the choice of line vs word as the recognition unit, and use of synthetic data to train the models. We compare our models with popular publicly available OCR tools for end-to-end document image recognition. Our end-to-end pipeline that employ our recognition models and existing text segmentation tools outperform these public OCR tools for 8 out of the 13 languages. We also introduce a new public dataset called Mozhi for word and line recognition in Indian language. The dataset contains more than 1.2 million annotated word images (120 thousand text lines) across 13 Indian languages. Our code, trained models and the Mozhi dataset will be made available at cvit-projects Additional Key Words and Phrases: indic ocr, indian languages, crnn, ctc, text recognition

Abstract

A system and method generating an optimized medical image using a machine learning model are provided . The method includes ( i ) receiving one or more medical images , ( ii ) segmenting to generate a transformed medical image for detecting a plurality of target elements , ( iii ) displaying the transformed medical image , ( iv ) receiving markings and scribblings associated with scribble locations from a user , ( v ) identifying errors associated with an outline of a target element , ( vi ) computing a loss function for a location of pixels where the target element is located on the transformed medical image , ( vii ) modifying the pre - defined weights ( w ) to match the segmentation output and the determined target element ,( viii ) determining whether the segmentation output is matched with the target element and ( ix ) generating the optimized medical image if the segmentation output is matched with the determined target element

Abstract

A system and method for generating an optimized image with scribble - based interactive image segmentation model using a machine learning are provided . The method includes , ( i ) segmenting , using a machine learning model , an image to classify into classes each class is represented with a label , ( ii ) displaying the classified image which specifies the classes on the classified image with outlines , ( iii ) enabling a user to scribble on the classified image to annotate the classes if an area is not classified , ( iv ) assigning a color mask for each scribbled area , ( v ) computing , using the machine learning model , a loss function for a location of pixels based on color mask , ( vi ) modifying pre - defined weights for each scribbled area to match the annotated image and a determined class on the classified image , and ( vii ) generating the optimized image if the annotated image is matched with the determined class on the classified image

Abstract

Keyword Spotting (KWS) detects a set of pre-defined spoken keywords. Building a KWS system for an arbitrary set requires massive training datasets. We propose to use the text transcripts from an Automatic Speech Recognition (ASR) system alongside triplets for KWS training. The intermediate representation from the ASR system trained on a speech corpus is used as acoustic word embeddings for keywords. Triplet loss is added to the Connectionist Temporal Classification (CTC) loss in the ASR while training. This method achieves an Average Precision (AP) of 0.843 over 344 words unseen by the model trained on the TIMIT dataset. In contrast, the Multi-View recurrent method that learns jointly on the text and acoustic embeddings achieves only 0.218 for out-of-vocabulary words. This method is also applied to low-resource languages such as Tamil by converting Tamil characters to English using transliteration. This is a very challenging novel task for which we provide a dataset of transcripts for the keywords. Despite our model not generalizing well, we achieve a benchmark AP of 0.321 on over 38 words unseen by the model on the MSWC Tamil keyword set. The model also produces an accuracy of 96.2% for classification tasks on the Google Speech Commands dataset. Index Terms: speech recognition, keyword spotting, lowresource languages