2.1.1. Supervised ML

Supervised learning refers to a class of algorithms that learn using data points with known outcomes/labels. The model is trained using an appropriate learning algorithm (such as linear regression, random forests, or neural networks) that typically works through some optimization routine to minimize a loss or error function. The model is trained by feeding it input data as well as correct annotations, whenever available. If the output of the model is continuous, this process is called regression and if the output is discrete with finite classes, the process is called classification. Supervised learning has been implemented for substantial applications in real life, such as image classification/segmentation, natural language processing, time-series analysis. Since supervised learning requires annotated data, it faces a bottleneck when labeling is expensive or laborious.

2.1.2. Unsupervised ML

In contrast to supervised learning, unsupervised learning involves models/algorithms that look for patterns in an unlabeled dataset with minimal human supervision. Some unsupervised learning paradigms include clustering, which aims to group data into similar categories (clusters); dimensionality reduction, which aims to find a low-dimensional subspace that captures most of the variation in the data; and generative modeling, which aims to learn the data distribution in order to generate synthetic data.

2.1.3. Reinforcement learning (RL)

RL is a type of agent-based ML where a complex system is controlled through actions that optimize the system in some manner (Sutton and Barto, Reference Sutton and Barto2018). The actions ( $ {a}_t $ ) taken based on a probability distribution $ {p}_{\pi } $ at a state ( $ {s}_t $ ) and time $ t $ seek to optimize the expected sum of rewards $ (r) $ based on a policy $ (J) $ parameterized by $ \theta $ ; i.e., $ J\left(\theta \right)={\sum}_{s_t,{a}_t\sim {p}_{\pi }}\left[r\left({s}_t,{a}_t\right)\right] $ .

RL is based on the Markov property of systems, where the evolution of system states is determined by the current state and the subsequent control sequence, and independent of the past states and past control inputs. Indeed, RL agents might be thought of as Markov decision processes (MDP) with neural nets serving as function approximators helping to decide which state transitions to attempt. RL is useful in when actions and environments are simple or data are plentiful. Early examples include backgammon (Tesauro, Reference Tesauro1994), the cart-pole problem, and Atari (Mnih et al., Reference Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra and Riedmiller2013). Recently, much work has been done to extend RL to broader uses.

RL architectures may be grouped in families with different learning methods; a simple one is policy gradient. Here, the controller is made up of a single function approximator that maps actions to states. The model is trained along the gradient of expected reward. One subfamily of policy gradient is the Actor-Critic that employs two function approximators; the actor estimates actions as in policy gradient, while the critic estimates the long-term value of the actions; these estimates contribute to the training of the actor network. An overview of the RL architecture taxonomy is provided in Figure 2.

Figure 2. A taxonomy of reinforcement learning architectures.

We now discuss specific classes of algorithms that are commonly used in the smart building domain.

2.2.1. Kernel-based methods

Kernel algorithms are designed for analyzing patterns in data from explicit feature vector representations provided to them. They work by projecting the onto hyperplanes that make the inference easier. Kernel methods can be used for supervised and unsupervised problems. A commonly used kernel-based method for classification is the support vector machine (SVM). Another common method in unsupervised settings is kernel principal component analysis (PCA), used for dimensionality reduction in order to facilitate clustering.

2.2.2. Ensemble methods

Ensemble methods combine multiple ML models to produce an optimal model. Common examples include tree-based algorithms such as decision tree, bagging (bootstrap aggregating), random forest, and gradient boosting.

2.2.3. Deep learning

Deep learning-based methods are usually deployed on extremely large datasets, and they use very complex artificial neural networks inspired by information processing in biological neurons for applications in supervised, semisupervised, or unsupervised settings. The neural network architecture depends on the nature of the available data. For example, convolutional neural networks typically analyze image data, while recurrent networks are usually applied to textual, natural language or time-series data.

2.2.4. Time-series forecasting methods

Time-series data comprise a sequence of data points indexed by time, such as stock prices, outdoor temperature, or electricity prices. Time-series analysis is used to extract statistical information about the data and potentially make predictions about future data points. Wide variety of methods can be used for time-series analysis and forecasting. Assuming future data have some relation with historically observed data, linear models such as moving average (MA), auto-regression (AR), and combinations of these can be used to forecast future data points. Some use cases are in stock price forecasting (Pai and Lin, Reference Pai and Lin2005) and in supply chain management (Aviv, Reference Aviv2003). Recurrent neural networks, and long short-term memory (LSTM) networks can also be used for time-series with more complex behaviors.

2.2.5. Physics-based ML

ML models have shown promising results in learning scientific problems that are yet to be well understood or where it is computationally infeasible with physics-based models. However, pure data-driven models often require very large datasets, offer limited extrapolation abilities, and are unable to provide physically consistent results or to offer intuition about the process they are modeling. On the other hand, pure physics-based models are often complex, include many uncertain parameters, and may be either too simplistic or too computationally intensive for the given application. This has led to the emergence of hybrid models that combine formal domain knowledge with ML. These hybrid models may compensate the weaknesses of pure physics-based models or pure data-driven models, and may be better able to both extrapolate and provide intuition about the process in question. Coupling physics knowledge with ML has been used in several areas, including medicine (Costabal et al., Reference Costabal, Yang, Perdikaris, Hurtado and Kuhl2020; Kissas et al., Reference Kissas, Yang, Hwuang, Witschey, Detre and Perdikaris2020), fluid mechanics (Cai et al., Reference Cai, Mao, Wang, Yin and Karniadakis2022), power systems (Misyris et al., Reference Misyris, Venzke and Chatzivasileiadis2020), manufacturing (Qi et al., Reference Qi, Chen, Li, Cheng and Li2019) etc. It has shown potential in better prediction accuracy using a smaller set of sample data and generalizability for out-of-sample scenarios.

2.2.6. Learning, dynamics, and control

Traditional control approaches such as system identification (Ljung, Reference Ljung1998) and adaptive control (Ioannou and Fidan, Reference Ioannou and Fidan2006) have similarities to supervised learning (Ljung et al., Reference Ljung, Andersson, Tiels and Schön2020) and RL (Matni et al., Reference Matni, Proutiere, Rantzer and Tu2019), respectively. While some system identification is implicit in all control approaches, as a part of how the model is formed, adaptive control never rose to the prominence of RL.

Several approaches to model-based RL use control-theoretic methods. Essentially, the problem is split into two parts: a dynamic model is learned from data, and the model is used to design a controller, with some variations, with model predictive control being a popular choice. Similarly, in traditional control applications, learned models are incorporated into modern control synthesis techniques, and standard methods of proving safety are adjusted to incorporate learned models. A learning-based approach that has proved to mesh particularly well with this approach uses Gaussian process models (Akametalu et al., Reference Akametalu, Fisac, Gillula, Kaynama, Zeilinger and Tomlin2014; Umlauft et al., Reference Umlauft, Lederer and Hirche2017; Wang et al., Reference Wang, Theodorou and Egerstedt2018; Devonport et al., Reference Devonport, Yin and Arcak2020, Reference Devonport, Yang, Ghaoui and Arcak2021b). Another point of contact is the use of statistical guarantees for controllers based on learned models. In RL, statistical guarantees of correctness such as regret bounds and probably approximately correct (PAC) bounds are used to ensure that optimal behavior is attained with high probability. By combining these statistical guarantees with control-theoretic techniques, they can also be used to ensure safety with high probability (Devonport and Arcak, Reference Devonport and Arcak2020; Devonport et al., Reference Devonport, Yang, El Ghaoui and Arcak2021a, Reference Devonport, Yang, Ghaoui and Arcak2021b). Since many learned models are statistical in nature, probabilistic bounds consistent with model accuracy have also been used extensively in recent years (Matni et al., Reference Matni, Proutiere, Rantzer and Tu2019).

3.3.1 Models

There are three types of building models: white box, black box, and gray box, as illustrated in Figure 5.

Figure 5. Three types of building models.

3.4.1. Model predictive control

MPC takes a large step toward improving these aims by incorporating optimization directly into the control policy. We wish to frame briefly our section on MPC: although it would not be considered a classic case of ML, we find it helpful to thoroughly embed the motivations behind its development and active use into our understanding of the learning-based methods to follow. Thus, we will spend significant time describing it.

The key idea of MPC is to predict the value of the objective function in the future using an auxiliary dynamical model of the building’s behavior and to select a control action that optimizes the predicted reward. Generally, if the dynamics model is $ f $ , the action $ u $ , the state $ x $ , the objective $ y $ , and the current timestep $ t $ , then MPC solves:

(3.1) $$ {\displaystyle \begin{array}{ll}\underset{u}{\hskip0.6em \mathrm{argmin}}& \sum \limits_{\tau =t}^{t+T}-y\left(x,u\right)\\ {}\mathrm{subject}\ \mathrm{to}:& x\left(\tau +1\right)=f\left(x\left(\tau \right),u\left(\tau \right)\right);\tau =t,\dots, T-1\\ {}& x\left(t+T\right)=0;\tau =t,\dots, T-1\\ {}& x\left(\tau \right)\in \mathcal{X};\tau =t,\dots, T-1\\ {}& u\left(\tau \right)\in \mathcal{U};\tau =t,\dots, T-1\\ {}& \end{array}} $$

The condition $ x\left(\tau \right)\mathcal{X} $ is a state-space constraint, which can be used to ensure safety by attempting to keep the state vector in a region of the state space that is known a priori not to be dangerous.

For a given state $ x $ , (3.1) computes a sequence of $ T $ control actions in order to minimize the predicted reward. While the building control policy could use all $ T $ actions, so that (3.1) need only be solved every $ T $ time steps, to do so would force the controller to ignore what is happening in the outside world for those time steps. Since the control model used in (3.1) can never be fully accurate, and since dynamical prediction errors compound very quickly, what happens during those time steps is likely very different from what the controller predicts in (3.1). Moreover, the controller would be effectively blind to unanticipated dangers while executing the sequence: by the time $ T $ time steps are up, it may be too late to bring the system back to safety.

Thus, the second key idea of MPC is to resist the temptation to use the full input sequence: an MPC controller solves (3.1) at each time step, and applies only the first computed input. Using only the first control action is how MPC mitigates potential dangers while retaining optimal behavior.

MPC has been widely used to achieve anticipatory control of HVAC system for energy saving while ensuring human comfort (Drgoňa et al., Reference Drgoňa, Arroyo, Figueroa, Blum, Arendt, Kim, Ollé, Oravec, Wetter, Vrabie and Helsen2020). MPC-based control methods require the solving of a multistep optimization problem so as to achieve the control sequence. The control output at the current stage is exerted and this process is repeated with the evolving of time. MPC-based control methods are online control methods that rely on the dynamic predictions of system disturbance. For HVAC control, several critical issues are related to the development of MPC-based controller: modeling complexity, modeling accuracy, and prediction accuracy. MPC-based controllers have been popular with HVAC control (see (Oldewurtel et al., Reference Oldewurtel, Parisio, Jones, Gyalistras, Gwerder, Stauch, Lehmann and Morari2012b; Afram and Janabi-Sharifi, Reference Afram and Janabi-Sharifi2014; Drgoňa et al., Reference Drgoňa, Arroyo, Figueroa, Blum, Arendt, Kim, Ollé, Oravec, Wetter, Vrabie and Helsen2020; Kathirgamanathan et al., Reference Kathirgamanathan, De Rosa, Mangina and Finn2021) for a comprehensive review).The MPC controllers generally use either physics-based models (also known as “analytical first principle” or “forward models”) or data-driven models (also known as “black box” or “inverse models”) to predict system output.

MPC-based controllers have many advantages over rule-based or PID controllers. Since the energy saving and human comfort targets may be directly expressed in the MPC’s objective and constraints and the time-varying disturbances are directly considered, MPC shows superior performance in energy efficiency and in ensuring human comfort. In addition, MPC-based controllers are robust to the time-varying disturbance, quick transient response to the environment changes, and consistent performance under a wide range of varying operating conditions.

However, MPC’s advantages are achieved at the cost of frequently solving comprehensive optimization problems online. The intrinsic problem related to the HVAC control is nonlinear and nonconvex. This leads to the high online computation cost. Many efforts have been made to address the computational challenges. One solution is to restrict attention to linear models of the form $ x\left(t+1\right)=A(t)x+B(t)u $ , where $ A(t) $ and $ B(t) $ are matrix functions whose values may be time-dependent but not state-dependent. This model simplification, in conjunction with a quadratic objective, reduces (3.1) to a quadratic program, a convex problem that even embedded devices can solve quickly. However, a simpler model will necessarily lead to predictions of lower accuracy, and thereby worse performance. If we do not wish to simplify the model, an alternative is to develop explicit MPC controllers (Drgoňa et al., Reference Drgoňa, Kvasnica, Klaučo and Fikar2013; Klaučo and Kvasnica, Reference Klaučo and Kvasnica2014; Parisio et al., Reference Parisio, Fabietti, Molinari, Varagnolo and Johansson2014). The key idea is to drive some close-form solution for the MPC problem and then identify the parameterized control outputs online. To achieve the objective, simplified or approximated linear models for HVAC control are generally used. The benefit of explicit MPC is that it can be implemented on simple hardware and yields low online computation cost. Another line of work to address the computation burden is to develop distributed solvers. When MPC-based controllers are applied to large-scale commercial buildings, we require to solve large-scale optimization problem that couples the control of the HVAC system supplying multiple zones or rooms. Centralized methods tend to suffer from the high computation cost and are not scalable. Many studies have made effort to develop distributed MPC controllers (Radhakrishnan et al., Reference Radhakrishnan, Su, Su and Poolla2016, Reference Radhakrishnan, Srinivasan, Su and Poolla2017; Zhang et al., Reference Zhang, Shi, Yan, Malkawi and Li2017; Yang et al., Reference Yang, Zhao, Li and Zomaya2020, Reference Yang, Srinivasan, Hu and Spanos2021). The major challenge to develop distributed MPC controllers for HVAC system is to handling the couplings across the multiple zones, including the thermal couplings (i.e., heat transfer), the composite objective (i.e., the fan power is quadratic with respect to the total zone mass flow rates), and operating constraints). The optimization problem related to multizone HVAC system is a nonlinear and nonconvex optimization problem subject to nonlinear couplings (nonlinear and nonconvex). The existing distributed solution methods cannot be directly applied to solve such a class of problems. To overcome the challenges and enable the decomposition of the problem, (Zhang et al., Reference Zhang, Shi, Yan, Malkawi and Li2017; Yang et al., Reference Yang, Zhao, Li and Zomaya2020, Reference Yang, Srinivasan, Hu and Spanos2021) has explored the convex relaxation technique to solve such problem. Particularly, Yang et al. (Reference Yang, Zhao, Li and Zomaya2020, Reference Yang, Srinivasan, Hu and Spanos2021) have applied the well-known alternating direction method of multipliers (ADMM) to solve the resulting relaxed problems, which have nonconvex objectives but subject to linear couplings. The superior performance both in energy saving and computation efficiency have been demonstrated via simulations. Slightly different, Radhakrishnan et al. (Reference Radhakrishnan, Su, Su and Poolla2016, Reference Radhakrishnan, Srinivasan, Su and Poolla2017) have explored the decomposition of the problem with respect to the zone controller via hierarchical optimization and relaxation. Particularly, these works have relied on predictions to predict the thermal couplings across the zones. Besides, the objective components are optimized in two steps. These studies have comprehensively discussed the challenges to develop distributed control methods for large-scale commercial buildings due to the problem complexity.

3.4.2. Reinforcement learning

Time-varying and uncertain parameters such as the weather variations and occupancy represent one major challenge for HVAC control. MPC controllers have relied on short-term dynamic predictions to address such problems but at the cost of high online computation burden and an accurate model to predict the system state evolution. These are the two main obstacles for the wide deployment of MPC controllers in practice. As one main branch of ML, RL explores to optimize the operation of system through the interaction with environment as long as the performance can be observed and quantified.

RL being one of the mainstream tools for multistage decision making under uncertainties (Sutton and Barto, Reference Sutton and Barto2018) makes it well suited for HVAC control. Indeed, RL has been widely used for advanced control of HVAC systems considering the uncertainties and multistage problem features (Sun et al., Reference Sun, Luh, Jia, Jiang, Wang and Song2012, Reference Sun, Luh, Jia and Yan2015; Yang et al., Reference Yang, Hu and Spanos2021). RL can enable efficient online computation by learning the control policies offline. Specifically, by learning the control policies offline, the online implementation of RL only requires to search the table to identify the mapping from the state space to the action space.

However, the obstacle to effective implementation of RL is that the learning may be computationally intensive or intractable when state and action space are large, either in terms of dimensionality (number of observations) or enumeration of action space (continuous vs. categorical.) To overcome the challenges, DRL has been popular in this domain. DRL proposes to combine the approximation and characterization capabilities of deep neural networks with RL techniques. Specially, deep neural networks are used to characterize the state or state action value functions. The benefits of DRL include reducing the quantity of data required for training and reducing the memory required to store the policy for online implementation. The proliferation of smart meters has boosted the development of DRL-based HVAC controllers for smart buildings. We direct the readers to (Han et al., Reference Han, May, Zhang, Wang, Pan, Yan, Jin and Xu2019; Mason and Grijalva, Reference Mason and Grijalva2019; Wang and Hong, Reference Wang and Hong2020; Yang et al., Reference Yang, Hu and Spanos2020; Yu et al., Reference Yu, Qin, Zhang, Shen, Jiang and Guan2021). As an earlier work, Wei et al. (Reference Wei, Wang and Zhu2017) has explored deep Q-network-based RL for single zone HVAC control. A Q-network is trained to approximate the state-value functions to overcome the computation burden related to the large state space. Yu et al. (Reference Yu, Xie, Xie, Zou, Zhang, Sun, Zhang, Zhang and Jiang2019) has explored the application of Deep Deterministic Policy Gradients (DDPG) (Lillicrap et al., Reference Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver and Wierstra2015) for the management of building energy system including an HVAC system. The benefit of DDPG is that it can enable the decision making on the continuous state and action space so as to improve the control accuracy. While most of the existing works have focus on the control of HVAC system for single zone or single room, several recent works have explored the development of distributed RL for multizone commercial buildings (Yu et al., Reference Yu, Sun, Xu, Shen, Yue, Jiang and Guan2020; Hanumaiah and Genc, Reference Hanumaiah and Genc2021). For example, Yu et al. (Reference Yu, Sun, Xu, Shen, Yue, Jiang and Guan2020) studied the application of multiagent actor-attention-critic RL (Iqbal and Sha, Reference Iqbal and Sha2019) for multizone HVAC control considering both thermal comfort and indoor air quality.

One major advantage of RL-based HVAC controllers over MPC is that it does not depend on explicit and accurate models, as in classical RL methods, a model of the environment may only ever be implicitly encoded in the inner layers of the policy or value prediction. In this way, RL methods alleviate the burden on the engineer for articulating an accurate model of the environment. It also allows for environment adaptation, i.e., flexibility to an environment that changes over time. The RL agent would just shift the value prediction it makes from actions over time.

RL methods are generally offline tools and require a rich set of data to compute the optimal control policies for a wide range of possible operation conditions. Moreover, the existing RL models including DRLs are still suffering from the computation intensity and scalability issues. Classical RL works best in cases where failure, at least at first, is relatively cheap. However, numerous studies elucidate ways that RL can be modified to reduce variability and increase safety. Arnold et al use Surprise Minimizing RL, a type of RL that rewards the agent for achieving states closer to those seen before, in optimizing building demand response (Arnold et al., Reference Arnold, Srivastava, Spangher, Agwan and Spanos2021). Augmenting RL with a system simulation is important, as almost always a large part of the training can take place in a simulation environment which can use a rules-based heuristic as a starting point, and use exogenous parameters to create a distribution of unique systems to train on (i.e., domain randomization). Finn firmly established the field of meta-learning in RL, arguing that a technique like model agnostic meta-learning can train in the different simulations to approximate a starting distribution for policy network weight initializations (Finn et al., Reference Finn, Abbeel and Levine2017). Jang et al. studies offline learning through offline-online RL (Jang et al., Reference Jang, Spangher, Khattar, Agwan and Spanos2021) and Model Agnostic Meta Learning (MAML) (Jang et al., Reference Jang, Spangher, Srivistava, Khattar, Agwan, Nadarajah and Spanos2021) to get RL agents “hit the ground ready” using offline datasets and simulations.

3.4.3. Learning-based MPC

From the above literature, we note that MPC and RL tools both have pros and cons for HVAC control. MPC-based controllers can yield superior performance both in energy saving and ensuring human comfort due to the capability to incorporate the dynamic predictions regarding the disturbances. However, this is achieved at the cost of high online computation burden. Moreover, MPC-based controllers generally suffer from the modeling complexity. For example, the more elaborate thermal comfort model, such as the PMV is hard to be incorporated in the MPC-based control due the challenge to solve the optimization problem (Xu et al., Reference Xu, Hu, Spanos and Schiavon2017). In contrast, RL-based models can accommodate the modeling complexities but require a huge amount of data to ensure the consistence of performance under uncertainties. The current research trends have been the combination of MPC and RL so as to enjoy both of their pros. The researches are mainly in the following two aspects. One is to use ML to mimic the control policies output by MPC controllers. Specifically, the control of MPC controllers can be regarded as the parameterized policy with respect to the disturbance. To reduce the online computation burden and enable the deployment of MPC controller on simple hardware, the characterization capacity of ML, such as deep neutral networks can be leverage to train parameterized control policies. This can yield control policies of high energy-saving performance and human comfort with MPC at lower online implementation cost. For example, Drgoňa et al. (Reference Drgoňa, Picard, Kvasnica and Helsen2018) proposed an approximate MPC where deep time delay neural networks (TDNN) and regression tree-based regression models are used to mimic the control policy of MPC controller. Simulation results show that the approximate MPC only incurs a slight performance loss (3%) but reduces the online computation burden by a factor of 7. Later, a real-time implementation of the approximate MPC is developed and demonstrated in Drgoňa et al. (Reference Drgona, Helsen and Vrabie2019) in an office building located in Hasselt, Belgium. Similarly, Karg and Lucia (Reference Karg and Lucia2018) proposed to use deep learning networks to learn the control policies of MPC controller to enable low-cost implementation. Using ML tools to learn the control policies of MPC controllers can be regarded as the extension of the explicit MPC discussed above. When the problem is simple and the closed-form solution is available, the implementation of explicit MPC only requires to compute the parameters of the parameterized policy and therefore the online implementation only requires low online computation cost. By leveraging ML tools, we are able to achieve the same objective by training the parameterized policies using deep neural networks when the problem is complex and the explicit closed-form solution is not accessible. Another line of work to combine ML with MPC is to use ML to establish the models for some complex components with building or HVAC system. We direct the readers to Afram et al. (Reference Afram, Janabi-Sharifi, Fung and Raahemifar2017) for a comprehensive review. For example, neural networks can be trained to efficiently predict PMV (Ferreira et al., Reference Ferreira, Ruano, Silva and Conceicao2012; Ku et al., Reference Ku, Liaw, Tsai and Liu2014) for a MPC controller. Since neural networks are generally characterized by nonlinear functions, the nonlinear solvers, such as genetic algorithms and particle swarm algorithms are often used to solve the resulting optimization problem. Compared with the original PMV models, neural network approximation can enable more efficient evaluation of the PMV index without requiring any iterations.

3.4.4. Summary

Though MPC and RL are two main tools for HVAC control, there has not been significant implementation of them in buildings, where the primary controllers have been PID and rule-based ones. There are still many practical issues required to be addressed to enable the deployment of MPC and RL-based methods. Challenges related to classic MPC controllers include the development of models that can properly trade off the modeling complexity and accuracy. In addition, the performance of MPC-based controllers is closely related to the prediction accuracy of disturbance and the prediction horizon. Last but not the least, MPC-based controllers suffer from the high online computation cost and the implementation requires advanced micro-controllers. In contrast, RL-based controllers are free from the establishment of models and can enable efficient and low-cost implementation but requires enormous data to train the agent. Moreover, RL-based controllers cannot incorporate the possibly available dynamic predictions to enhance the performance. That is why most of the existing RL-based controllers are not comparable with MPC-based controllers in terms of energy-saving performance. Another critical issue related to MPC controller is that the safely cannot be ensured. Since the agent is trained on limited data, RL-based controller may draw the system to any possible state which may be prohibitive in practice. Combining ML with MPC-based controllers seems a promising solution as it can enjoy both the pros of ML and MPC. However, how well the control policies can be learned and how the network should be designed remain to be investigated. The operation of HVAC system is expected to provide a comfortable indoor environment for the occupants.

3.4.5. Mechanism design

Compliance is the ability for people, in aggregate, to follow guidelines or recommendations. In any system that seeks to influence human behavior, lack of compliance is generally a large problem. Energy systems are no different: an empirical study of demand response found that noncompliance weakened the overall effect of the programs (Cappers et al., Reference Cappers, Goldman and Kathan2010).

Gamification of systems, i.e., the adding of an additional layer of intrinsic motivation to a core program, may be an important tool to boost compliance. The additional layer of motivation may come in the form of engagement-derived pleasure (i.e., interesting and exciting gameplay), competition (i.e., equally matched competitors), or social status (i.e., badges, ranks, and advancement). Hamari et al., Reference Hamari, Koivisto and Sarsa2014 rightly note the explosion in interest in gamification, from gamification around code learning to gamification around energy demand reduction and perform an extensive review of its efficacy, finding that generally gamification provides positive results. Figure 6 provides a visualization of the gamification pipeline and the purpose it serves.

Figure 6. An example of a system in which gamification can be part of a control strategy.

There are some examples of gamifying office-related energy systems in the literature. Although not extensively, ML algorithms have been utilized in the gamification process to improve some individual components of the game. In one example, a game called “NTU Social Game” has been formulated around reductions in energy in a residential dormitory. Here, players compete against each other to have the lowest energy, and accrue points depending on how much less energy they consume. Prizes are not simply doled out according to whoever has the highest energy reductions; instead, players are entered into a prize pool if they are among the top third performing participants. A small number of prizes are allocated randomly among the winning pool as per the Vickrey Clarke Groves (VCG) auction mechanism. A design like this is shown to boost user engagement and directive compliance (Konstantakopoulos et al., Reference Konstantakopoulos, Barkan, He, Veeravalli, Liu and Spanos2019a, Reference Konstantakopoulos, Das, Barkan, He, Veeravalli, Liu, Manasawala, Lin and Spanos2019b). The NTU Social Game has been used for energy reduction and to test different visualizations on player outcome (Spangher et al., Reference Spangher, Tawade, Devonport and Spanos2019). It has also been used to conduct segmentation analysis of players into low, medium, and high energy-efficient groups for intelligent incentive planning (Das et al., Reference Das, Konstantakopoulos, Manasawala, Veeravalli, Liu and Spanos2019, Reference Das, Konstantakopoulos, Manasawala, Veeravalli, Liu and Spanos2020). The authors use a hybrid approach of supervised classification and unsupervised clustering, to create the characteristic clusters that satisfy the energy efficiency level. They also have incorporated explainability in the model by combining both the ML approaches.

An extension of this work, “OfficeLearn”, considers prices that are distributed across a day. Here, a building controller sets energy prices throughout the day, and players compete with each other on having the lowest cost of energy throughout the day. As before, the top performing individuals are entered into a prize pool and selected based on a VCG auction mechanism (Spangher et al., Reference Spangher, Gokul, Khattar, Palakapilly, Tawade, Bouyamourn, Devonport and Spanos2020). This paper presents an OpenAI gym environment for testing demand response with occupant level building dynamics, useful for standardizing RL algorithms implemented for the same purpose.

Other examples of energy-based games exist. In one game called “energy chickens”, competitors save energy from smart appliances for the health of virtual chickens; depending on how healthy the chickens are, they will generate probabilistically more or less eggs (Orland et al., Reference Orland, Ram, Lang, Houser, Kling and Coccia2014). In another game, “Energy Battle”, players in simulations of identical residential homes strive to reduce energy and players compete one to one without a VCG mechanism (Geelen et al., Reference Geelen, Keyson, Boess and Brezet2012). In a third game, “EnerGAware”, players also compete in a building simulation in which players, who are virtual cats, decide how to operate a house’s appliances with greater energy efficiency while not reducing any overall functions (Casals et al., Reference Casals, Gangolells, Macarulla, Forcada, Fuertes and Jones2020). The use of ML in energy social games, although not prominent before, is showing signs of increase, and is adequately justified since ML can have a profound use in coordinating multiagent systems such as a game.

3.5.1. Demand response

We start off by discussing how individual buildings can use their load flexibility as DR resources. DR is a broad term for a variety of methods used to module electricity demand. These can include load response to time-varying electricity rates, financial incentives, or direct control of loads (like appliances and air-conditioning) by utilities.

3.5.2. DR aggregations

Load flexibility can be procured through utility programs as discussed previously, and also through direct market participation by loads. There has been a move in recent years to allow loads to submit bids for load reduction into wholesale electricity markets, and treat those bids similar to generation bids. Federal regulations, specifically Federal Energy Regulatory Commission (FERC) Order 2222 in the United States have mandated that distributed resources be allowed to participate in wholesale markets. The state of California has instituted a market mechanism called DR auction mechanism to allow loads to submit bids as ‘negative generation’. However, the magnitude of load flexibility provided by individual buildings is often too small for them to participate directly in electricity markets. This is where aggregations can play a role: aggregators can bundle the load modulation capability of multiple buildings and use the aggregate capacity to submit bids to electricity markets. DR can also be coupled with other resources like generation and storage, and form a part of a virtual power plant.

3.5.3. Prosumers and local energy markets

So far, we covered how buildings can use their load flexibility to provide services to the power grid, both individually and through aggregations. Additionally, buildings can directly consume energy from onsite clean resources. Smart buildings are often equipped with a variety of generation and storage resources such as rooftop solar, battery backups, and electric vehicle chargers, which can be used to reduce demand charges or optimize grid consumption. They can profit from these resources by supplying energy back to the grid, i.e., functioning as prosumers. A prosumer is an entity that is capable of energy production as well as consumption through the presence of local generation and energy storage devices.