Abstract

One of the major drawbacks of RL is the low sample efficiency of the learning algorithms. In many cases domain expertise can help to mitigate this effect. Teacher-Student framework is one such paradigm, where a more experienced agent (teacher) upon being queried helps to accelerate the student’s learning by providing advice on the action to take in a given state. Real world teachers not only provide the action to take in a given state but also provide a more informative signal using the synthesis of knowledge they may have gained with experience. With this motivation, we propose a richer advising framework where the teacher augments the student’s knowledge by also providing the expected long term reward of following that action. The student can then use this value to steadily guide its Q-Network in the correct direction which can lead to a quicker convergence. To help student relive the advices received throughout its learning, we introduce an additional memory called the Advice Replay Memory (ARM). Results show that a student following our approach (a) is able to exploit the environment better, and (b) has a steeper learning curve.

Abstract

We describe the design of an autonomous electricity broker agent, VidyutVanika, the runner-up of the 2018 PowerTAC competition. The agent uses techniques from reinforcement learning, dynamic programming and other areas of machine learning to seek appropriate actions in tariff and wholesale market of the PowerTAC simulation environment. The novelty of our agent lies in defining the reward functions of suitably defined Markov decision processes (MDPs), solving these MDPs, and applying these solutions to real actions in the market. In addition, VidyutVanika uses a neural network to predict the energy consumption of various customers using weather data. The usage forecasts, so obtained, are used to place orders in day-ahead wholesale market. These forecasts also helps in reducing the balancing costs incurred by the broker.

Abstract

A smart grid is an efficient and sustainable energy system that integrates diverse generation entities, distributed storage capacity, and smart appliances and buildings. A smart grid brings new kinds of participants in the energy market served by it, whose effect on the grid can only be determined through high fidelity simulations. Power TAC offers one such simulation platform using real-world weather data and complex state-of-the-art customer models. In Power TAC, autonomous energy brokers compete to make profits across tariff, wholesale and balancing markets while maintaining the stability of the grid. In this paper, we design an autonomous broker VidyutVanika, the runner-up in the 2018 Power TAC competition. VidyutVanika relies on reinforcement learning (RL) in the tariff market and dynamic programming in the wholesale market to solve modified versions of known Markov Decision Process (MDP) formulations in the respective markets. The novelty lies in defining the reward functions for MDPs, solving these MDPs, and the application of these solutions to real actions in the market. Unlike previous participating agents, VidyutVanika uses a neural network to predict the energy consumption of various customers using weather data. We use several heuristic ideas to bridge the gap between the restricted action spaces of the MDPs and the much more extensive action space available to VidyutVanika. These heuristics allow VidyutVanika to convert near-optimal fixed tariffs to time-of-use tariffs aimed at mitigating transmission capacity fees, spread out its orders across several auctions in the wholesale market to procure energy at a lower price …

Abstract

Periodic Double Auctions (PDAs) are commonly used in the real world for trading, eg in stock markets to determine stock opening prices, and energy markets to trade energy in order to balance net demand in smart grids, involving trillions of dollars in the process. A bidder, participating in such PDAs, has to plan for bids in the current auction as well as for the future auctions, which highlights the necessity of good bidding strategies. In this paper, we perform an equilibrium analysis of single unit single-shot double auctions with a certain clearing price and payment rule, which we refer to as ACPR, and find it intractable to analyze as number of participating agents increase. We further derive the best response for a bidder with complete information in a single-shot double auction with ACPR. Leveraging the theory developed for single-shot double auction and taking the PowerTAC wholesale market PDA as our testbed, we proceed by modeling the PDA of PowerTAC as an MDP. We propose a novel bidding strategy, namely MDPLCPBS. We empirically show that MDPLCPBS follows the equilibrium strategy for double auctions that we previously analyze. In addition, we benchmark our strategy against the baseline and the state-of-the-art bidding strategies for the PowerTAC wholesale market PDAs, and show that MDPLCPBS outperforms most of them consistently.

Abstract

Decentralized (PO) MDPs provide a rigorous framework for sequential multiagent decision making under uncertainty. However, their high computational complexity limits the practical impact. To address scalability and real-world impact, we focus on settings where a large number of agents primarily interact through complex joint-rewards that depend on their entire histories of states and actions. Such history-based rewards encapsulate the notion of events or tasks such that the team reward is given only when the joint-task is completed. Algorithmically, we contribute---1) A nonlinear programming (NLP) formulation for such event-based planning model; 2) A probabilistic inference based approach that scales much better than NLP solvers for a large number of agents; 3) A policy gradient based multiagent reinforcement learning approach that scales well even for exponential state-spaces. Our inference and RL-based advances enable us to solve a large real-world multiagent coverage problem modeling schedule coordination of agents in a real urban subway network where other approaches fail to scale.

Abstract

Traffic management during emergency evacuation throws up a different set of challenges than a regular traffic management. We focus here on one particular challenge, namely prioritized routing. Prioritized routing may need to happen even during normal times but stands out during emergencies, since emergency vehicles, police vehicles and vehicles (such as buses) that carry a lot more people may need to have a higher priority in terms of evacuation. It is also reasonable to assume that traffic police may need to perform a centralized control of traffic since they typically have a global view of the emergency and possibly have accurate real-time updates. We therefore make the following contributions in this paper: (a) We map the prioritized routing problem to the minimum-cost maximum-flow problem, a standard problem formulation in network flow theory. (b) We then develop the Prioritized Routing Assistant for Flow …

Abstract

The Cooperative Target Observation (CTO) problem has been of great interest in the multi-agents and robotics literature due to the problem being at the core of a number of applications including surveillance. In CTO problem, the observer agents attempt to maximize the collective time during which each moving target is being observed by at least one observer in the area of interest. However, most of the prior works for the CTO problem consider the targets movement to be Randomized. Given our focus on surveillance domain, we modify this assumption to make the targets strategic and present two target strategies namely Straight-line strategy and Controlled Randomization strategy. We then modify the observer strategy proposed in the literature based on the K-means algorithm to introduce five variants and provide experimental validation. In surveillance domain, it is often reasonable to assume that the observers may themselves be a subject of observation for a variety of purposes by unknown adversaries whose model may not be known. Randomizing the observers actions can help to make their target observation strategy less predictable. As the fifth variant, we therefore introduce Adjustable Randomization into the best performing observer strategy where the observer can adjust the expected loss in reward due to randomization depending on the situation.

Abstract

Humans display a variety of behaviors including greedy and other unexpected biases during emergency situations. In this paper, we study the effect of such behaviors on traffic patterns during an emergency situation. In particular,we assume that during an emergency there are traffic police or other personnel trying to guide the vehicles so that the traffic clearance is sped up (to minimize the human or property losses). However each individual driver agent has its own motives and biases that may slowdown such a clearing up of traffic. In particular, we assume that the traffic police use the Ford-Fulkerson Algorithm to maximize the flow in the (traffic) network and then perform an extensive agent based analysis to study the traffic patterns that arise due to a variety of human factors and biases. Through a series of experiments performed using the well-known traffic simulator SUMO we could show that: (a) Ford-Fulkerson Algorithm is indeed efficient for clearance of vehicles in such settings (b)Driver agents that do not follow the directions due to their preferences and biases lose out on an average and (c) As can be expected, having more information about the environment available via prior knowledge and/or real-time updates can offset the effect of biases to a good extent.

Abstract

Slow moving traffic in heavily populated cities, can many times result in loss of lives due to emergency vehicles not being able to reach their destination hospitals on time. Recent advances in the field of Intelligent Transportation Systems (ITS) makes it increasingly likely that vehicles in the near future will be equipped with advanced systems that allow inter vehicular communication. In this paper, we assume the usage of such a system to optimize the lane level dynamics for an emergency vehicle (EV), traversing a multi lane stretch of road under a variety of traffic settings. In particular, we present the Fixed Lane Strategy (FLS) and the Best Lane Strategy (BLS) for EV traversal and perform an extensive agent based analysis to study their strengths and weaknesses. Through a series of experiments performed using the well-known traffic simulator SUMO, we could show that: (a) BLS performs better than SUMO strategy on all traffic settings we tested. (b) BLS performs better than FLS in settings that capture real-world traffic conditions involving congestion and uncertainties while FLS performs better in well-behaved conditions and (c) BLS was found to be the best strategy for the setting calibrated using real world data (obtained from NYCDOT).

Abstract

Slow moving traffic in heavily populated cities, can many times result in loss of lives due to emergency vehicles not being able to reach their destination hospitals on time. In this paper, we assume the usage of inter vehicular communication to optimize the lane level dynamics for an Emergency Vehicle Agent (EVA), traversing a multi lane stretch of road in an urban traffic setting. In particular, we present the Fixed Lane Strategy (FLS) and the Best Lane Strategy (BLS) for EVA traversal. FLS is a simple strategy that acts as a good baseline while BLS is a sophisticated strategy that can adapt to varying traffic patterns.