Abstract

The widespread access to the Internet has led to a meteoric rise in audio-visual content consumption. Our content consumption habits have changed from listening to podcasts and radio broadcasts to watching videos on YouTube. We are now increasingly preferring the highly engaging nature of video calls over plain voice calls. Given this considerable shift in desire for audio-visual content, there has also been a surge in video content creation to cater to these consumption needs. In this fabric of video content creation, especially those containing people talking, lies the problem of making these videos accessible across language barriers. If we want to translate a deep learning lecture video in English to Hindi, it is not only that the speech should be translated but also the visual stream, specifically, the lip movements. Learning to lip-sync arbitrary videos to any desired target speech is a problem with several applications ranging from video translation, to readily creating new content that would otherwise require humongous efforts. However, speaker-independent lip synthesis for any voice, and language is a very challenging task. In this thesis, we tackle the problem of lip-syncing videos in the wild to any given target speech. We propose two new models in this space: one that significantly improves the generation quality and the other significantly improving on lip-sync accuracy. In the first model, LipGAN, we identify key issues that plague the current approaches for speakerindependent lip synthesis that prevent them from reaching the generation quality of speaker-specific models. Specifically, ours is the first model to generate face images that can be pasted back into the video frame. This feature is crucial for all the real-world applications where the face is just a small part of the entire content being displayed. We show that our improvements in quality lead to multiple real-world applications that have not been demonstrated in any of the previous lip-sync works. In the second model, Wav2Lip, we investigate why current models are inaccurate while lip-syncing arbitrary talking face videos. We hypothesize that the reason is weak penalization. This finding allows us to create a lip-sync model that can generate lip-synced videos for any identity and voice with remarkable accuracy and quality. We re-think the current evaluation framework for this task and propose multiple new benchmarks, two new metrics, and a Real world lip Sync Evaluation Dataset (ReSyncED). Also, using our model, we show applications on lip-syncing dubbed movies and animating real CGI movie clips to new speech. We also demonstrate a futuristic video call application that is useful for poor network connections. Finally, we present two major appli cations that our model can impact the most social media content creation and personalization and video translation. We hope that our advances in lip synthesis open up new avenues for research in the space of talking face generation from speech.

Abstract

The recent surge of data and trends in machine learning and deep learning helped to increase our understanding of language and thereby resulting in many applications. But the same is not true for many under resourced languages. Telugu is one such low resource Indian language. This work aims to shed light on the research of Telugu language. We present corpora and their analysis in the areas of Conversational Dialogue Systems, Named Entity Recognition and Emotion Prediction respectively. We present a Telugu conversational corpus, the first ever corpus to the best of our knowledge. We have built an end-to-end dialogue system using the corpus and performed various experiments with sequence to sequence encoder and attention decoder model involving word order, translation, vocabulary size, transliteration and word representations. The second is a Telugu-English code-mixed social media corpus for Named Entity Recognition(NER), the first ever corpus to the best of our knowledge. We have experimented with traditional machine learning methods such as Conditional Random Fields(CRFs), an undirected probabilistic graphical model, Decision Trees and also using a deep learning method, Long Short term Memory Networks(LSTMs). We have proposed feature functions for Named Entity Recognition which were used in the CRF. We reported an F1-score of 0.96, 0.94 and 0.95 with CRFs, Decision Trees and Bidirectional LSTMS respectively. The third is a Telugu-English code-mixed social media corpus for Emotion prediction, the first ever corpus to the best of our knowledge. We have proposed feature functions for Emotion Prediction which were used in an experiment with Support Vector Machines(SVM). We have also experimented with deep learning methods such as Long Short term Memory Networks(LSTMs) and Bidirectional LSTMs. SVM, LSTMs and Bidirectional LSTMs reported an accuracy of 58%, 60.92% and 70.74% respectively

Abstract

The aim of this research is to cultivate, nurture and establish the formal foundation of arts and their relations in association with time and space. Primarily we begin with the question that, why there exist so many forms of art? What constitutes the existence of their plural beings? How are they formally distinct from each other? And what are the formal features which connect them as well. We start with the concepts of forms and language in arts and gradually move towards conceiving the following four chapters in the thesis. Chapter 1. Origin and Foundations of Arts Chapter 2. Idea of Form, Mind, Body, and Truth Chapter 3. Blending of Sound and Vision: Study of Rāgamālā Paintings in Association with the North Indian Classical Music Chapter 4. Blending of Site and Architecture: Installation as a Form of Art The idea is to examine a creative mind by re-visiting the rational space of each art form in terms of their formal purity and origin. We use the term ‘purity’ or ‘a pure being’in the sense of exactness, something which is apt, just that and nothing else. It refers to the qualitative existence of a being and doesn’t imply any moral ground. Any work of art necessarily requires actions to move from one space to another in the imaginary horizon. It’s a kind of force for any constructive creation. Is there any creation without construction? We claim that there is no true creation without having a construction. And here comes the question of rationality. To be more specific, the rational being of every art form. Form itself is saṅskār-rahit or devoid of saṅskār-s. But any formal action is necessarily saṅskār- hit or born out of saṅskāra-s. There is an idea of ‘muse’ in the Greek mythology which refers to the spirit of arts. ‘Muse’-s and spirits are plural beings. But how and where does one get the spirit-s/ muse-s from? We instigate that the spirit- s/muse-s are born out of saṅskāra-s. Saṅskāras are sheltered in the cultivated memories. The cultivated memories are born out of repetitive act-s/ habit-s/ ādab-s etc. It gives birth of skills.The skills for imagination, judgement, re-creation and transformation in the process of creating a work of art to investigate the nature of reality beyond its physical existence. The nature of art construes the growth of a creative mind. And the nature of its construal is completely a mental phenomenon until it gets transformed into the material existence. Art can be purely a mental phenomenon. But any work of art necessarily requires a material body. And that’s how it gets visibility in the real world. So the act of giving a material body to the ideas of any creative mind needs to be examined to validate its formal identity and the nature of its transformal quality at the same time. Work of art is a re-creation or anukṛti of imagination and memory. The act of its mechanism involves a rational structure irrespective of conscious or sub-conscious state of any creative mind. We study and examine the fundamentals of arts in connection with their origins and foundations in the first chapter of our thesis. It’ll falsify the notion that the act of an artistic creation is irrational

Abstract

The Indian IT industry is being continuously disrupted by rapidly improving automation technologiesthat tend towards replacing repetitive IT support work and back-office tasks previously driven by humanactors. This increase in efficient automation-based technologies leverages software to carry out back-office tasks almost independently of human- where foreign organizations originally invested in. Thetrend of automating manual work practices in the industry has generated extreme debates over theirprojected influence on hiring patterns, employment strategies, job profiles, work practices, and the overalleconomy. Using a socio-technical framework, the thesis aims at investigating the manifold consequencesarising out of introducing automation techniques in the software Quality Testing (QT) segment of theIndian IT industry. Ethnographic methods were employed for the collection of qualitative researchdata, to identify key socio-economic factors influencing the decision to automate a testing environment.The resulting discourse has been viewed through an Information and Communications Technology forDevelopment (ICTD) lens, by primarily analyzing first-person responses of Quality Testers injectingvalue into the testing job profile and colligating these discourses with existing QT practices. Subsequently,the significance of designing improved QT work practices in combination with relevant skill-trainingstrategies has been detailed, to enable IT professionals to exploit their implicit knowledge, businessprocess knowledge, critical thinking, and computing skills while collaborating with automated QTprocedures.The second part of the thesis focuses on the reskilling and up-skilling required to take the edge off jobthreats in the IT industry owing to work automation. India is home to the largest under-25 demographicprofile in the world, requiring a widespread, skill-oriented educational model, equipping youth to thrivein highly dynamic job markets. As a response to the huge demand for technical education, a large privateskill tutoring ecosystem has sprung up in Ameerpet, Hyderabad, India, whose pedagogical model can beadopted by Massive Open Online Courses (MOOCs) to improve their underlying foundation of teachingtechniques. The thesis offers reasons as to why learners are inclined to enroll in the traditional physicalmodels of Ameerpet classrooms over online training courses. This further draws attention towards designrecommendations that can be incorporated in online learning platforms to entice learners who often getmarginalized in the more formal and competitive education system and opt for Ameerpet-like skillinghubs. The principal offering in this context would be to propose a shift in the perspective of MOOCsto include job readiness by escorting suggestions in course curriculum, course delivery, and marketingstrategies

Abstract

Livestreaming platforms enable content producers or streamers to broadcast creative content to a potentially large viewer base. Chatrooms form an integral part of such platforms, enabling viewers to interact both with streamer and amongst themselves. Streams with high engagement (many viewers and high active chatters) are typically considered engaging and often promoted to end users by means of recommendation algorithms, and exposed to better monetization opportunities via revenue share from platform advertising, viewer donations and third-party sponsorships. Given such incentives, some streamers make use of fraudulent means to increase perceived engagement by simulating chatter via fake “chatbots” which can be purchased from online marketplaces. This inorganic engagement can negatively influence recommendations, hurt streamer and viewer trust in the platform, and harm monetization for honest streamers. In this study, we tackle the novel problem of automating detection of chatbots on livestreaming platforms. To this end, we first formalize the livestreaming chatbot detection problem and characterize differences between botted and genuine chatter behaviour observed from a real-world livestreaming chatter dataset collected from Twitch.tv. We then propose SHERLOCK and BOTHUNT methods, which posits a two-stage approach of detecting chatbotted streams, and subsequently detecting constituent chatbots. Finally, we demonstrate effectiveness on both real and synthetic data: to this end, we propose a novel strategy for collecting labeled, synthetic chatter dataset (typically unavailable) from such platforms, enabling evaluation of proposed detection approaches against chatbot bahaviors with varying signatures. The SHERLOCK approach achieves 97% precision/recall on the real world dataset and +80% F1 score across most simulated attack settings and BOTHUNT achieves 86% accuracy for real world dataset and 93% accuracy across all attack settings. This thesis is a timely contribution to the area of computer science specially combating astroturfing, needed to mitigate the spread of fraudulent bot users on Live streaming Platforms. The results from this thesis can be used to build real world solutions to mitigate the spread of untrustworthy or botted streams, fake users, etc. on live streaming platforms in the future.

Abstract

We are intrigued by the problem of Web Accessibility. While it is somewhat intuitive to imagine the barriers a disabled person may face in accessing content on the web, they are not the only segments of users facing accessibility challenges. For instance, users with poor English abilities, or users with less domain exposure, or users with different thinking levels also face access barriers when reaching out to mainstream targeted public content. English-only text for the non-English users, technical jargon for the non-technical, scholarly material for the novice user, and local cultural references to those uninitiated may also be seen as barriers to accessibility. To such non-mainstream users,who are often in the minority, the content on the web – which is often designed for the majority – could, at times, be perceived as complex, foreign, incomprehensible and even inaccessible. In our work, we are focused on this aspect of the Web Accessibility problem. We adopt the existing social technique of Renarration as a mechanism to overcome the accessibility problem. At a social level, when humans communicate with a diverse group of listeners, they vary the content to suit the needs of differing communities of their users. For example, a child who is trying to discuss a new mod1 in a game may elaborate it more for a fellow gamer, but may instead simplify it for a parent. Similarly, a scholar may formalize an explanatory text for a fellow scholar, but may simplify or abstract the same to suit the needs of a conventional listener. We use the term renarration to describe this act of generating a variant version of the content from an original source. More formally, renarration can be treated as a semantic transformation function applied on a given web page to generate a variant. In our work, we have chosen to overcome some of the accessibility barriers on the web by applying the technique of renarration to it. And, we equate renarration of web content to a semantic transformation function. And finally, we aspire to find a web architecture that would facilitate such semantic transformation of web content. Given this, the problem statement for our research can now be stated as: What is that web architecture that would enable one or more volunteer users to manipulate, modify, process or transform published web content into one or more alternate views, such that each variant view now may deliver a different meaning to its targeted community of viewers? We move towards the solution space by reviewing the current approaches that are already in use, and also by exploring any earlier work from the e-publishing industry. The design of our solution is further informed by a series of nearly two dozen research undertakings consisting of multiple surveys and experiments. As part of our research, we also explore the theoretical underpinnings for semantic transformation. We study the notion of ”semantics” (as applied in Computer Science literature), and also the notion of ”transformations” as discussed in the mathematical disciplines of geometry and algebra. Through this work, we capture our notion of semantic transformation of web content in an abstract, symbolic notation. Citing ease of use for non-technical users, we focus on the notion of style sheets as one of the many ways to transform web content. As part of this study, we explore the general notion of style sheets and see how specific style sheets – namely, the popular XSLT and CSS style sheets – have been used on the web. We take an abstract view of CSS as a transformation function t operating on HTML marked content c and augmented by style information s, resulting in t = f(c,s). Our notion of semantic transformation is informed by this abstraction and is represented as yet another semantic transformation function t = f(c 0 , l, a, n) where f is still a transformation function, c 0 is now the semantically annotated source, l is the location and description of the semantic structure imposed upon the page by the user, a is the new set of actions that are inclusive of semantics, n represents the conditions that are to be applied, and t is the transformed target page. We call this new semantically oriented style sheet as Semantic Style Sheet or SSS. We take the core ideas behind CSS rule-set and use that to inform the design of our SSS. For instance, the notion of a rule-set underlying a CSS is given by selector : { property : value}. We enhance this rule-set in three dimensions, and infer a simplistic rule-set representation for SSS, which is given as locator : { ( action , condition ) } where {attribute : value} pairs are predefined for each of the locator, actions and conditions parameters. To automate the better instrumentation of a web page we found the conventional tree metaphor inadequate. We further explore other conceptual representations of a web page and present an ontological model as our r

Abstract

In this research work, we present three syntax-based computational methods that use the dependency structure of sentences for inference generation, summarization, and machine reading comprehension. Besides, we present an introductory dataset to aid the production of inferences from a response in a context. The methods described in this thesis utilize the dependency structure obtained from the parser that represents the dependency relations existing between the words of the sentence. This work explains how these relations are utilized to compute inferences from utterances, summarize documents, and answer questions based on context passages. The first method describes computing pragmatic inferences from news headlines that are stand-alone units of text carrying no context information. The different types of pragmatic inferences are entailment, presupposition, and implicature. News headlines are some minimal utterances that contain maximum information for which the speaker is the news editor, and the receiver is the newsreader. The inferences, such as presuppositions and conventional implicatures that are independent of the context, depend mainly on syntax. Hence, this approach uses dependency trees of the headlines to find the syntactic structure and to compute inferences out of them. The generated inferences about the news headlines could be useful for assessing the impact of the same on readers, including children. Utterances do not frequently exist in isolation, like in news headlines. Hence, we created a minimal dataset with implicated meanings of utterances in a context. In this work, 1k context-response utterance dialogue pairs having implicatures for the response utterance are collected and annotated with their implicated meanings. Since dialogues with implicatures are not easy to obtain, the data collection methods, challenges, findings, and sources of dialogue data of this genre are also explained. This dataset and further additions to it would improve the understanding capability of the machine and reduce the amount of conversational failures in human-computer interactions. The second method describes summarizing a passage based on syntax and dependency structure. The extractive summary of the passage is obtained by finding overlapping sentences from the dependency graph constructed by attaching the dependency trees of individual sentences in the passage and simultaneously prioritizing content word nodes such as verbs, nouns, adjectives, and adverbs, and discarding translative nodes such as prepositions and articles. The third method uses dependency trees to simulate a human approach while answering a question about a passage. While reading a passage to answer a question, humans usually try to identify the verb of action in the question and find the sentences with the same or synonymous verbs as the candidate answers. Simulating the human approach of finding the same or similar verb in the passage comparing to the verb in the question is followed for comprehending the passage to answer a question. The verb nodes of the dependency trees are compared to achieve this.

Abstract

In the era of structural genomics, with large number of protein structures becoming available, identification of domains is an important problem in protein function analysis as it forms the first step in protein classification. Domain identification has been an active area of research for over four decades and wide range of automated methods have been proposed. In the proposed network-based machine learning approach, NML-DIP, a combination of supervised (SVM) and unsupervised (k-means) machine learning techniques are used for domain identification in proteins. The algorithm proceeds by first representing protein structure as a protein contact network and using topological properties, viz., length, density and interaction strength (that assesses inter- and intra-domain interactions) as feature vectors in the first SVM to distinguish between single and multi-domain proteins. A second SVM is used to identify the number of domains in multi-domain proteins. Thus, it does not require a prior information of the number of domains. The k-means algorithm is then used to identify the domain boundaries that are assessed using CATH annotation. Performance of the proposed algorithm is evaluated on four benchmark datasets and compared with four state-of-the-art domain identification methods. The performance of the approach is comparable to other domain identification tools and works well even when the domains are formed with non-contiguous segments. The performance of the program is significantly improved for prior information about the number of domains. The algorithm is available at: https://bit.ly/NML-DIP.

Abstract

Novels are a mine of information stored in a narrative. Detective novels usually have a premise of a culprit and a detective consolidating the evidence. The key ingredient in a detective novel is obfuscation of the evidence so that the reader cannot easily determine the culprit. We formulate a computational problem here where we identify sentences containing potential evidence, if any, against each character in the novel. While novels and stories from other genres have a more linear narrative, detective novels reference information from different points in the story. In this genre, the author is motivated to reveal only partial information to a reader to keep them engaged. So first, we propose a method to gather evidence against every character by extracting relevant sentences which seem to implicate them in the crime and are scattered throughout the story. Thus by identifying evidence sentences against each character, we computationally generate an ‘evidence summary‘. We split the novel into pre and post culprit expose by identifying the reveal paragraph, and utilise the information post culprit expose to identify semantically similar sentences in their vector representations. We process each novel text as their graph representation, and employ different graph traversal techniques. Second, from the exercise of extracting evidence we identify and categorise three major ways in which popular authors in crime and detection genre embed evidence in their stories. We compare our algorithm results with human evaluators and qualitatively determine the different writing styles. Our results show we outperform the baseline method in extracting useful evidence sentence and categorising novels on the basis of clue obfuscation. For one of books in the dataset The Murder of Roger Ackroyd, our method obtains a score of 8/10 compared to a 3/10 baseline. Thus our contributions are two fold - we propose a method to extract evidence containing sentences, in a detective novel utilising the semantics of this genre, and also establish different ways in which authors obfuscate the evidence.

Abstract

Internet of things (IoT) devices are steadily becoming mainstream in our lives and simplifying themanner in which we perform everyday tasks. With billions of smart devices, embedded with all kinds oftransducers, communicating with each other and us humans via the internet, data is being generated atan alarming rate. Consequently, data analytics has become an integral part of the IoT ecosystem. SeveralIoT applications, including those that come under the umbrella of smart home, intelligent transportationsystem, smart healthcare and smart grid, rely on machine learning (ML) to gain inference from sensordata and aid in decision making. Apart from user applications, ML is also used in the lower layers,including the network layer and the medium access control (MAC) layer, to improve the efficiency ofIoT networks.The focus of this thesis is on the application of ML in making IoT systems smarter and more effi-cient. More specifically, a new ML-based paradigm for occupancy estimation is introduced along withan ML-based data transmission reduction scheme that is validated for occupancy applications. Havingknowledge of the occupancy status of rooms can help in automating lighting systems, cooling/heatingsystems as well as a myriad of other appliances throughout the building. The proposed occupancyestimation paradigm employs ML algorithms on a deployment of multiple non-intrusive sensor nodesin a room. Two new datasets were created in the process which are thoroughly investigated by ap-plying various preprocessing, feature engineering and ML techniques. Multiple performance metricslike accuracy, F1 score and confusion matrix are reported for a variety of homogeneous and heteroge-neous feature combinations. A novel ML-based data transmission reduction scheme is also proposed forapplication-specific IoT networks that takes the application into account when deciding the initiation oftransmission. Wireless data transmission is infamous for being the most energy consuming activity ofa sensor node. Therefore, intelligently reducing data transmissions can prolong the lifetime of battery-powered sensor nodes without comprising much on the data quality. To enable the proposed scheme forconstrained sensor nodes, the complexities of various ML algorithms are discussed, both theoreticallyand practically, from the perspective of constrained microcontrollers. One of the occupancy estimationdatasets along with a standard dataset for occupancy detection are used for validating the proposed datatransmission reduction scheme. Experimental results demonstrate a humongous reduction in the num-ber of data transmissions while upholding similar performance metrics. The proposed scheme is alsoshown to significantly outperform the Shewhart change detection algorithm for occupancy applications