Abstract

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems, and more, optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. We study the set of existing differentially-private RL policies derived from various RL algorithms such as Value Iteration, Deep-Q Networks, and Vanilla Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL analysis framework (PRIL) that involves performing reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using the Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using this framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on instances of the FrozenLake domain of varying complexities. Based on the analysis performed, we infer a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We do so by quantifying the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards.

Abstract

An autonomous broker that liaises between retail customers and power-generating companies (GenCos) is essential for the smart grid ecosystem. The effciency brought in by such brokers to the smart grid setup can be studied through a well-developed simulation environment. In this paper, we describe the design of one such intelligent energy broker called VidyutVanika21 (VV21) and analyze its performance using a simulation platform called PowerTAC (Power Trading Agent Competition). Specifcally, we discuss the retail (VV21–RM) and wholesale market (VV21–WM) modules of VV21 that help the broker achieve high net profts in a competitive setup. Supported by game-theoretic analysis, the VV21–RM designs tariff contracts that a) maintain a balanced portfolio of different types of customers; b) sustain an appropriate level of market share, and c) introduce surcharges on customers to reduce energy usage during peak demand times. The VV21–WM aims to reduce the cost of procurement by following the supply curve of the GenCo to identify its lowest ask for each auction which is then used to generate suitable bids. We further demonstrate the effcacy of the retail and wholesale strategies of VV21 in PowerTAC 2021 fnals and through several controlled experiments.

Abstract

We present a Nash equilibrium analysis for single-buyer singleseller multi-unit 𝑘-double auctions for scaling-based bidding strategies. We then design a Deep Deterministic Policy Gradient (DDPG) based learning strategy, DDPGBBS, for a participating agent to suggest bids that approximately achieve the above Nash equilibrium. We expand DDPGBBS to be helpful in more complex settings with multiple buyers/sellers trading multiple units in a Periodic Double Auction (PDA), such as the wholesale market in smart-grids. We demonstrate the efficacy of DDPGBBS with Power Trading Agent Competition’s (PowerTAC) wholesale market PDA as a testbed.

Abstract

When the Indian government declared the frst lockdown on 25 March 2020 to control the increasing number of COVID-19 cases, people were forced to stay and work from home. The aim of this study is to quantify the impact of stay-at-home orders on residential Air Conditioning (AC) energy and household electricity consumption (excluding AC energy). This was done using monitored data from 380 homes in a group of fve buildings in Hyderabad, India. We gathered AC energy and household electricity consumption data at a 30-min interval for each home individually in April 2019 and April 2020. Descriptive and inferential statistical analysis was done on this data. To ofset the diference in temperatures for the month of April in 2019 and 2020, only those weekdays were selected where the average temperature in 2019 was same as the average temperature in 2020. The study establishes that the average number of hours the AC was used per day in each home increased in the range 4.90–7.45% depending on the temperature for the year 2020. Correspondingly, the overall AC consumption increased in the range 3.60–4.5%, however the daytime (8:00 AM to 8:00 PM) AC energy consumption increased in the range 22–26% and nighttime (8:00 PM to 8:00 AM) AC energy consumption decreased by 5–7% in the year 2020. The study showed a rise in household electricity consumption of about 15% for the entire day in the year 2020. The household electricity consumption increased during daytime by 22- 27.50% and 1.90- 6.6% during the nighttime. It was observed that the morning household electricity peak demand shifted from 7:00 AM in 2019 to 9:00 AM in 2020. Conversely, the evening peak demand shifted from 9:00 PM in 2019 to 7:00 PM in 2020. An additional peak was observed during afternoon hours in the lockdown.

Abstract

When the Indian government declared the frst lockdown on 25 March 2020 to control the increasing number of COVID-19 cases, people were forced to stay and work from home. The aim of this study is to quantify the impact of stay-at-home orders on residential Air Conditioning (AC) energy and household electricity consumption (excluding AC energy). This was done using monitored data from 380 homes in a group of fve buildings in Hyderabad, India. We gathered AC energy and household electricity consumption data at a 30-min interval for each home individually in April 2019 and April 2020. Descriptive and inferential statistical analysis was done on this data. To ofset the diference in temperatures for the month of April in 2019 and 2020, only those weekdays were selected where the average temperature in 2019 was same as the average temperature in 2020. The study establishes that the average number of hours the AC was used per day in each home increased in the range 4.90–7.45% depending on the temperature for the year 2020. Correspondingly, the overall AC consumption increased in the range 3.60–4.5%, however the daytime (8:00 AM to 8:00 PM) AC energy consumption increased in the range 22–26% and nighttime (8:00 PM to 8:00 AM) AC energy consumption decreased by 5–7% in the year 2020. The study showed a rise in household electricity consumption of about 15% for the entire day in the year 2020. The household electricity consumption increased during daytime by 22- 27.50% and 1.90- 6.6% during the nighttime. It was observed that the morning household electricity peak demand shifted from 7:00 AM in 2019 to 9:00 AM in 2020. Conversely, the evening peak demand shifted from 9:00 PM in 2019 to 7:00 PM in 2020. An additional peak was observed during afternoon hours in the lockdown. Keywords: COVID-19, Residential energy consumption, Air conditioning energy consumption, Household electricity consumption, Case study, Data monitoring, India

Abstract

When the Indian government declared the frst lockdown on 25 March 2020 to control the increasing number of COVID-19 cases, people were forced to stay and work from home. The aim of this study is to quantify the impact of stay-at-home orders on residential Air Conditioning (AC) energy and household electricity consumption (excluding AC energy). This was done using monitored data from 380 homes in a group of fve buildings in Hyderabad, India. We gathered AC energy and household electricity consumption data at a 30-min interval for each home individually in April 2019 and April 2020. Descriptive and inferential statistical analysis was done on this data. To ofset the diference in temperatures for the month of April in 2019 and 2020, only those weekdays were selected where the average temperature in 2019 was same as the average temperature in 2020. The study establishes that the average number of hours the AC was used per day in each home increased in the range 4.90–7.45% depending on the temperature for the year 2020. Correspondingly, the overall AC consumption increased in the range 3.60–4.5%, however the daytime (8:00 AM to 8:00 PM) AC energy consumption increased in the range 22–26% and nighttime (8:00 PM to 8:00 AM) AC energy consumption decreased by 5–7% in the year 2020. The study showed a rise in household electricity consumption of about 15% for the entire day in the year 2020. The household electricity consumption increased during daytime by 22- 27.50% and 1.90- 6.6% during the nighttime. It was observed that the morning household electricity peak demand shifted from 7:00 AM in 2019 to 9:00 AM in 2020. Conversely, the evening peak demand shifted from 9:00 PM in 2019 to 7:00 PM in 2020. An additional peak was observed during afternoon hours in the lockdown. Keywords: COVID-19, Residential energy consumption, Air conditioning energy consumption, Household electricity consumption, Case study, Data monitoring, India

Abstract

Electricity markets are getting more dynamic and complex today. On the one hand, there are outside-in factors like new entrants and increasing competition, regulatory mandates, and volatility of the electricity prices. In contrast, on the other hand, there are inside-out factors like the adoption of renewable energies, storages, and EV, emphasis on the generation and retail portfolio optimization together with creating affinity services around the empowered consumers. The paper will start with the introduction of this dynamic environment across the globe with a specific focus on Australia, the UK, and US electricity markets. Introduction of 5 min settlement in Australian Electricity Market, the rise of ancillary service products in the UK, or new roles like Community Choice Aggregators (CCA) in the US are some of the

Abstract

The teacher-student framework aims to improve the sample efficiency of RL algorithms by deploying an advising mechanism in which a teacher helps a student by guiding its exploration. Prior work in this field has considered an advising mechanism where the teacher advises the student about the optimal action to take in a given state. However, real-world teachers can leverage domain expertise to provide more informative signals. Using this insight, we propose to extend the current advising framework wherein the teacher would provide not only the optimal action but also a qualitative assessment of the state. We introduce a novel architecture, namely Advice Replay Memory (ARM), to effectively reuse the advice provided by the teacher. We demonstrate the robustness of our approach by showcasing our experiments on multiple Atari 2600 games using a fixed set of hyper-parameters. Additionally, we show that a student taking help even from a suboptimal teacher can achieve significant performance boosts and eventually outperform the teacher. Our approach outperforms the baselines even when provided with comparatively suboptimal teachers and an advising budget, which is smaller by orders of magnitude. The contributions of our paper are 4-fold (a) supplementing student’s knowledge by providing the state category (b) introduction of ARM to effectively reuse the advice throughout learning (c) ability to achieve significant performance boost even with a coarse state categorization (d) enabling the student to outperform the teacher.

Abstract

In modular reinforcement learning (MRL), a complex decision making problem is decomposed into multiple simpler subproblems each solved by a separate module. Often, these subproblems have conflicting goals, and incomparable reward scales. A composable decision making architecture requires that even the modules authored separately with possibly misaligned reward scales can be combined coherently. An arbitrator should consider different module’s action preferences to learn effective global action selection. We present a novel framework called GRACIAS that assigns fine-grained importance to the different modules based on their relevance in a given state, and enables composable decision making based on modern deep RL methods such as deep deterministic policy gradient (DDPG) and deep Q-learning. We provide insights into the convergence properties of GRACIAS and also show that previous MRL algorithms reduce to special cases of our framework. We experimentally demonstrate on several standard MRL domains that our approach works significantly better than the previous MRL methods, and is highly robust to incomparable reward scales. Our framework extends MRL to complex Atari games such as Qbert, and has a better learning curve than the conventional RL algorithms.

Abstract

Stackelberg games have found a role in a number of applications including modeling market competition, identifying traffic equilibrium, developing practical security applications and many others. While a number of solution approaches have been developed for these games in a variety of contexts that use mathematical optimization, analytical analysis or heuristic based solutions, literature has been quite sparse on the usage of Genetic Algorithm (GA) based techniques. In this paper, we develop a GA based solution to compute high quality mixed strategy solution for the leader to commit to, in a General Stackelberg Game (GSG). Our experiments showcase that the GA solution developed here indeed performs well in terms of scalability and provides reasonably good solution quality in terms of the average reward obtained.