Abstract

One of the major drawbacks of RL is the low sample efficiency of the learning algorithms. In many cases domain expertise can help to mitigate this effect. Teacher-Student framework is one such paradigm, where a more experienced agent (teacher) upon being queried helps to accelerate the student’s learning by providing advice on the action to take in a given state. Real world teachers not only provide the action to take in a given state but also provide a more informative signal using the synthesis of knowledge they may have gained with experience. With this motivation, we propose a richer advising framework where the teacher augments the student’s knowledge by also providing the expected long term reward of following that action. The student can then use this value to steadily guide its Q-Network in the correct direction which can lead to a quicker convergence. To help student relive the advices received throughout its learning, we introduce an additional memory called the Advice Replay Memory (ARM). Results show that a student following our approach (a) is able to exploit the environment better, and (b) has a steeper learning curve.

Abstract

Decentralized MDPs (Dec-MDPs) provide a rigorous framework for collaborative multi-agent sequential decisionmaking under uncertainty. However, their computational complexity limits the practical impact. To address this, we focus on a class of Dec-MDPs consisting of independent collaborating agents that are tied together through a global reward function that depends upon their entire histories of states and actions to accomplish joint tasks. To overcome scalability barrier, our main contributions are:(a) We propose a new actor-critic based Reinforcement Learning (RL) approach for event-based Dec-MDPs using successor features (SF) which is a value function representation that decouples the dynamics of the environment from the rewards;(b) We then present Dec-ESR (Decentralized Event based Successor Representation) which generalizes learning for event-based Dec-MDPs using SF within an end-to-end deep RL framework;(c) We also show that Dec-ESR allows useful transfer of information on related but different tasks, hence bootstraps the learning for faster convergence on new tasks;(d) For validation purposes, we test our approach on a large multi-agent coverage problem which models schedule coordination of agents in a real urban subway network and achieves better quality solutions than previous best approaches.

Abstract

Traffic management during emergency evacuation throws up a different set of challenges than a regular traffic management. We focus here on one particular challenge, namely prioritized routing. Prioritized routing may need to happen even during normal times but stands out during emergencies, since emergency vehicles, police vehicles and vehicles (such as buses) that carry a lot more people may need to have a higher priority in terms of evacuation. It is also reasonable to assume that traffic police may need to perform a centralized control of traffic since they typically have a global view of the emergency and possibly have accurate real-time updates. We therefore make the following contributions in this paper: (a) We map the prioritized routing problem to the minimum-cost maximum-flow problem, a standard problem formulation in network flow theory. (b) We then develop the Prioritized Routing Assistant for Flow …

Abstract

Decentralized (PO) MDPs provide a rigorous framework for sequential multiagent decision making under uncertainty. However, their high computational complexity limits the practical impact. To address scalability and real-world impact, we focus on settings where a large number of agents primarily interact through complex joint-rewards that depend on their entire histories of states and actions. Such history-based rewards encapsulate the notion of events or tasks such that the team reward is given only when the joint-task is completed. Algorithmically, we contribute---1) A nonlinear programming (NLP) formulation for such event-based planning model; 2) A probabilistic inference based approach that scales much better than NLP solvers for a large number of agents; 3) A policy gradient based multiagent reinforcement learning approach that scales well even for exponential state-spaces. Our inference and RL-based advances enable us to solve a large real-world multiagent coverage problem modeling schedule coordination of agents in a real urban subway network where other approaches fail to scale.

Abstract

The Cooperative Target Observation (CTO) problem has been of great interest in the multi-agents and robotics literature due to the problem being at the core of a number of applications including surveillance. In CTO problem, the observer agents attempt to maximize the collective time during which each moving target is being observed by at least one observer in the area of interest. However, most of the prior works for the CTO problem consider the targets movement to be Randomized. Given our focus on surveillance domain, we modify this assumption to make the targets strategic and present two target strategies namely Straight-line strategy and Controlled Randomization strategy. We then modify the observer strategy proposed in the literature based on the K-means algorithm to introduce five variants and provide experimental validation. In surveillance domain, it is often reasonable to assume that the observers may themselves be a subject of observation for a variety of purposes by unknown adversaries whose model may not be known. Randomizing the observers actions can help to make their target observation strategy less predictable. As the fifth variant, we therefore introduce Adjustable Randomization into the best performing observer strategy where the observer can adjust the expected loss in reward due to randomization depending on the situation.

Abstract

Slow moving traffic in heavily populated cities, can many times result in loss of lives due to emergency vehicles not being able to reach their destination hospitals on time. In this paper, we assume the usage of inter vehicular communication to optimize the lane level dynamics for an Emergency Vehicle Agent (EVA), traversing a multi lane stretch of road in an urban traffic setting. In particular, we present the Fixed Lane Strategy (FLS) and the Best Lane Strategy (BLS) for EVA traversal. FLS is a simple strategy that acts as a good baseline while BLS is a sophisticated strategy that can adapt to varying traffic patterns.

Abstract

Slow moving traffic in heavily populated cities, can many times result in loss of lives due to emergency vehicles not being able to reach their destination hospitals on time. Recent advances in the field of Intelligent Transportation Systems (ITS) makes it increasingly likely that vehicles in the near future will be equipped with advanced systems that allow inter vehicular communication. In this paper, we assume the usage of such a system to optimize the lane level dynamics for an emergency vehicle (EV), traversing a multi lane stretch of road under a variety of traffic settings. In particular, we present the Fixed Lane Strategy (FLS) and the Best Lane Strategy (BLS) for EV traversal and perform an extensive agent based analysis to study their strengths and weaknesses. Through a series of experiments performed using the well-known traffic simulator SUMO, we could show that: (a) BLS performs better than SUMO strategy on all traffic settings we tested. (b) BLS performs better than FLS in settings that capture real-world traffic conditions involving congestion and uncertainties while FLS performs better in well-behaved conditions and (c) BLS was found to be the best strategy for the setting calibrated using real world data (obtained from NYCDOT).

Abstract

Humans display a variety of behaviors including greedy and other unexpected biases during emergency situations. In this paper, we study the effect of such behaviors on traffic patterns during an emergency situation. In particular,we assume that during an emergency there are traffic police or other personnel trying to guide the vehicles so that the traffic clearance is sped up (to minimize the human or property losses). However each individual driver agent has its own motives and biases that may slowdown such a clearing up of traffic. In particular, we assume that the traffic police use the Ford-Fulkerson Algorithm to maximize the flow in the (traffic) network and then perform an extensive agent based analysis to study the traffic patterns that arise due to a variety of human factors and biases. Through a series of experiments performed using the well-known traffic simulator SUMO we could show that: (a) Ford-Fulkerson Algorithm is indeed efficient for clearance of vehicles in such settings (b)Driver agents that do not follow the directions due to their preferences and biases lose out on an average and (c) As can be expected, having more information about the environment available via prior knowledge and/or real-time updates can offset the effect of biases to a good extent.