3.1. Building modeling

To perform accurate optimization, the system needs access to realistic building measurements and the ability to make accurate predictions of future states. Prior works have carefully studied energy and air quality monitoring by utilizing sensor nodes and accessing building management systems. However, thermal comfort as an objective is more challenging, as it requires either direct measurement of individuals or indirect modeling using environmental sensing. Indirect modeling is not always accurate and does not take into account personal thermal preferences. Thermal comfort sensing, on the other hand, has uncovered a number of challenges such as perspective and scalability.

Once the building state can be sensed, various models can be used to make predictions of future states. These predictions are critical to exploring the possible future solution space in optimization. However, standard computational methods for modeling commercial buildings such as EnergyPlus (Crawley et al., Reference Crawley, Lawrie, Winkelmann, Buhl, Huang, Pedersen, Strand, Liesen, Fisher, Witte and Glazer2001) tend to perform poorly in comparison to real measurements (Norford et al., Reference Norford, Socolow, Hsieh and Spadaro1994). In the context of energy simulation in our use case, standard physics-based tools like EnergyPlus experience a few key challenges. These models are typically evaluated at the facility level, making it difficult to understand differences in energy consumption across individual spaces or rooms in buildings. These models also tend to exhibit larger errors as temporal granularity increases, making it difficult to understand how short-term interventions impact energy. Finally, these models have been shown to experience difficulty in handling occupancy data, one of the key features that impact energy consumption and form a key recommendation strategy in our overall system. To address these challenges, researchers have pointed to data-driven surrogate modeling, typically leveraging supervised machine learning tools, as an alternative approach that can address the key issues with physics-based modeling outlined here. As discussed below, we implement a novel simulation environment based on data-driven statistical modeling to address these challenges associated with simulating different building states.

3.2. Co-optimizing energy, comfort, and air quality

Typical methods for building optimization, such as energy consumption, include changing setpoints in different locations. The effects of this are reductions in energy but also changes to thermal comfort and air quality. Various works have found that changing setpoints, and thus the room temperatures while keeping within thermal comfort boundaries, can reduce energy consumption. In this work, we also seek to optimize thermal comfort and air quality; thus, this strategy may not be optimal. We take a different approach by incorporating humans in the co-optimization process. We add an additional kind of recommendation, a move action, to move people from one location to another. To illustrate the potential advantages of the move recommendation, we provide the following example.

Consider two occupants A and B, in two different rooms 1 and 2, respectively. Without move recommendations, the best that can be done is to change the setpoint temperatures, which may reduce energy consumption at the cost of affecting thermal comfort and air quality. However, with move actions, occupant A can be recommended to move to room 2, and the setpoint of room 1 can be changed to reduce energy consumption.

The main challenge is how to determine the actions, and sequences of actions, which can result in optimization of the different metrics. Furthermore, emphasis on certain metrics may be more desired depending on the situation. Air quality may be a focus during a pandemic, while thermal comfort may be a focus in office environments where worker productivity is highly valued.

3.3. Deep reinforcement learning recommendation systems

Sequences of actions can lead to more optimal building states which cannot be estimated from single actions. In this work, we utilize deep reinforcement learning to learn the impacts of these sequences of actions; however, there are a number of challenges that limit the application of standard deep reinforcement learning techniques to building co-optimization of multiple objectives. First, energy, comfort, and air quality are all objectives which can be co-optimized. While prior works co-optimizing multiple objectives combine the objectives into a single reward, this creates a few issues. The importance of the different objectives may be different between people and information is lost about individual objective changes that can be presented to the occupants.

Second, our goal is to incorporate human actions into the co-optimization process. To do this, we require human location as part of the building state. However, the possible location configurations of occupants are a scaling problem for large numbers of people or rooms. For example, representing the locations of individuals as one hot vector results in a state space that is $ {\left|R\right|}^{\mid O\mid } $ , where R and O are the sets of rooms and occupants, respectively. To enable scaling to larger deployments, a network capable of incorporating location information in a dense manner for learning can lead to a better understanding of the state space.

Finally, training a deep network requires a large amount of training data. In this application, important training data consists of energy savings, comfort improvements, and air quality improvements due to certain actions, which are low in volume. To overcome this cold start problem, we require the ability to quickly generate training data that can reasonably estimate objective changes for different actions depending on the environment.

4.1. Deep reinforcement learning formulation

We represent the building co-optimization problem as follows. At each time step, the network uses a policy to choose an action a from a set of possible actions $ a\in A $ based on the current state of the building, s, which constantly changes over time due to agent recommendations and external factors like outside temperature. This action is sent to the actor(s) (occupants in our case), which is then accepted or rejected. Our system then sees a reward, r, representing short-term changes given by the environment, which would be in improvements energy, comfort, and air quality. For example, our system may recommend the only occupant in room A to move to room B. The occupant accepts and moves to room B, allowing the building to turn down the HVAC and lights in room A. As a result, the system observes a reward of energy savings from room A.

The building environment continuously changes, due to agent recommendations and external factors such as occupant location changes and environmental factors. Thus, we can formulate the problem as a finite Markov Decision Process (MDP). Each action has a transition probability of occurrence at the current state and a reward ( $ r $ ), representing short-term changes due to the action.

The goal of the deep Q-network is to estimate long-term changes associated with each action given the current building state s. Standard reinforcement learning aims to maximize this return at time t, defined as $ {R}_t={\sum}_{i=t}^T{\gamma}^i{r}_i $ , where r_i is the reward observed at time i, $ \gamma \in \left(0,1\right) $ is the “discount factor”, and T is the end time (or end of day in commercial buildings). We leverage Q-learning, a widely-used model-free reinforcement learning method, where the agent seeks an action-value function Q(s,a) which represents the return of taking an action a at a given state s. If the state/action space is too large, Q(s,a) is too complex to be stored in a data structure, but can be approximated effectively using a deep neural network Mnih et al. (Reference Mnih, Kavukcuoglu, Silver, Graves, Antonoglou, Wierstra and Riedmiller2013)).

4.1.2. Reward

Since we are co-optimizing energy savings, thermal comfort, and air quality, the reward is the improvement in these objectives from one time step to the next and depends on the current building state, current action, and the building state at the beginning of the next time step. Equation (1) shows the reward at the timestep n.

(1) $$ {r}_n=-\alpha {E}_n-\beta {C}_n-\gamma {Q}_n $$

E_n refers to the total energy consumption of all energy-consuming resources, C_n refers to the total comfort of all occupants, and Q_n is the total air quality rating experienced by all occupants in the building at timestep n. Higher values of E_n, C_n, and Q_n correspond to higher energy consumption, lower overall occupant comfort, and lower overall air quality experienced by occupants. $ \alpha $ , $ \beta $ , $ \gamma $ are used as weights for the three objectives, allowing building managers to customize and select which objective(s) to prioritize. Next, we describe how we compute E_n, C_n, and Q_n.

1. Energy consumption: E_n is computed in Equation (2), where $ \Delta $ is the length of one-time step and $ {P}_d(t) $ is the power consumption of the energy-consuming resource d (e.g., HVAC and lights). E_n is therefore the total energy consumed by all energy-consuming resources d in the nth time window.

(2) $$ {E}_n={\int}_{t=n\Delta}^{\left(n+1\right)\Delta}\sum \limits_d{P}_d(t) dt. $$

2. Thermal comfort: C_n is computed in Equation (3), where $ C\left({R}_o,t\right) $ is the comfort of occupant o at location R_o at time t. As such, C_n is the total comfort across all occupants in the building at timestep n. Measuring comfort is challenging, and we discuss how we measure comfort in our real deployments in Section 5.

(3) $$ {C}_n=\frac{\sum_{o\in O}{\int}_{t=n\Delta}^{\left(n+1\right)\Delta}C\left({R}_o,t\right) dt}{\Delta \cdot \mid O\mid } $$

3. Air quality: Q _n is computed as shown in Equation (4). $ Q\left({R}_o,t\right) $ is the air quality experienced by occupant o at location R_o at time t. Much like thermal comfort, the reward only considers the air quality of the areas where occupants are present because the air quality of empty rooms will not affect any of the occupants. We discuss how we measure the air quality of different rooms, locations, and spaces in Section 5.

(4) $$ {Q}_n=\frac{\sum \limits_{o\in O}{\int}_{t=n\Delta}^{\left(n+1\right)\Delta}Q\left({R}_o,t\right) dt}{\Delta \cdot \mid O\mid } $$

4.2. Deep Q-network for generating actionable recommendations

There are two challenges unique to building co-optimization that prevent direct application of a standard deep Q-network. The first challenge is representing occupant locations in a building. Typically, categorical data such as occupant locations are represented using one-hot encoding. However, the number of input nodes in the neural network quickly increases with the number of occupants and rooms. We address this challenge by incorporating an embedding layer (Section 4.2.1). The second challenge is the representation of the reward; there are many actions in each state and three different objectives (energy, comfort, and air quality) for each action. The network must learn the objectives for all states and actions. We address this challenge through multitask learning (Section 4.2.2).

4.2.1. Location embedding

One important observation of occupant locations is that there are hidden relationships in the encoding of the occupant locations. Consider an occupant who spends time in three different locations: two similar office spaces and one lab space. Let us assume that the energy consumption, setpoint temperature, humidity and air quality are similar for the two office spaces. A one-hot encoding of the locations will not uncover similarities between these spaces and will have to be learned through a number of layers.

One method for learning these similarities between spaces is to use an embedding layer, which has a few key advantages. First, embedding layers reduce the input size of the model by eliminating the need for a one-hot encoding. This can help reduce computation time, thus reducing training time. Second, an embedding layer has been proven to learn relationships between categories (Rong, Reference Rong2014), which can lead to improved learning. In our deep Q-network, we utilize an embedding layer to better learn the occupant location configurations than standard dense layers.

Instead of representing each occupant’s location as a one-hot vector, each location is assigned a unique numerical value corresponding to a row in the embedding layer. The embedding layer selects the rows corresponding to the location vector for each occupant, which are concatenated into a dense vector of size $ \mid O\mid \times d $ , where d is the embedding dimension and $ \mid O\mid $ is the number of occupants. This output can be a more informative input for future layers by encoding the relationships between locations. An illustration of the embedding layer is shown in Figure 2. In our deployments, we observe that d = 3 yielded the best tradeoff in performance vs computation.

Figure 2. The embedding layer in our network converts the physical representation of occupant locations into a feature representation through a learned embedding matrix.

4.2.3. Network architecture

The network input consists of the sparse location features of each occupant and dense features, including temperature, humidity, setpoint temperature, air quality, and energy consumption for each location. The complete network architecture is shown in Figure 3. The sparse location features are fed into the embedding matrix, and the output vectors are flattened and concatenated with the dense features. The new vector is fed to the shared layers, and then to the individual objective task layers. The output of each task is values representing the expected change in the energy consumption ( $ {E}_n^a $ ), comfort ( $ {C}_n^a $ ), and air quality ( $ {Q}_n^a $ ) objectives for all possible actions, a, at time step n.

Figure 3. Our deep Q-network architecture includes an embedding layer for learning occupant locations and has three output tasks for learning energy savings, comfort and air quality improvements for each action.

4.3. Recommender system design

The upper half of Figure 1 shows the full online system, consisting of the following components: the networked sensing layer (Section 5) that senses the building state (Section 4.1.1), the deep-Q network model (Section 4.2) and the ranker (Section 4.3.1) that uses the building state to generate recommendations for occupants, and the occupant-facing web client (Section 4.3.2) where users can view and accept/reject recommendations.

4.3.1. Ranker

Once the Q-network estimates the changes in energy consumption, comfort, and air quality objectives, the ranker arranges the recommendations for each occupant to maximize the overall energy savings, comfort, and air quality improvements. To accomplish this, the ranker generates a score for each action, a, according to Equation (5).

(5) $$ {S}_n^a=-\alpha {E}_n^a-\beta {C}_n^a-\gamma {Q}_n^a $$

$ \alpha $ , $ \beta $ , and $ \gamma $ are weights that building managers can tune to prioritize certain objectives. There is a negative sign with each objective because a higher value is less desirable (Sections 4.1 and 5). This objective is identical to the reward of the system (Equation 1) and essentially allows the building manager to control and bias the system based on which objectives s/he emphasizes. In Section 7.4, we compare improvements in each objective using different weights in real deployments.

As RECA recommends two different categories of recommendations directly to users (move and setpoint), it is important to give the occupants a selection from both categories (diversity). The ranker selects two recommendations from each category that are displayed to the user. The ranker uses softmax action selection to select actions to recommend users at time step $ n $ , after normalizing scores, $ {S}_n^a $ , for all actions, $ a\in A $ , using the softmax function (Equation 6).

(6) $$ p\left(a,n\right)=\frac{\exp \frac{S_n^a}{\tau }}{\sum_{b\in A}\exp \frac{S_n^a}{\tau }} $$

$ \tau $ is a temperature parameter. Sampling recommendations this way also allows RECA to incorporate exploration. Softmax selection has one key advantage over $ \unicode{x025B} $ -greedy methods, which select the action with the best-expected return with probability $ 1-\unicode{x025B} $ (exploitation) and a random action with probability $ \unicode{x025B} $ (exploration). During exploration, when $ \unicode{x025B} $ -greedy methods choose a random action, the actions with the worst expected return have the same probability of being chosen as the actions with the best-expected return. If some actions have zero probability of being performed (such as a move recommendation to a location that is not acceptable to the occupant), choosing this action over another action that may have a slightly lower return than the best return will slow training significantly. Softmax selection weights the probability of selecting an action during exploration based on its expected return, which reduces the slowdown effect of low-return actions during training.

4.3.2. User feedback

To serve recommendations to occupants in real-time, we developed a web interface (Figure 4) where occupants can browse a list of up-to-date recommendations, and select recommendations which are sent back to the system and stored in history as feedback. Additionally, we store the history of observed building states in a digital twin, similar to the one presented in Srinivasan et al. (Reference Srinivasan, Manohar and Issa2020). In Section 6, we discuss how we use feedback and the history of building states to address several challenges for deploying RECA in the real world.

Figure 4. The web interface for occupant feedback displays a list of recommendations. Estimated energy savings, comfort, and air quality improvements are shown with each recommendation.

6.1. Digital twin

Digital twins provide a virtual representation of the real-time building state. Since each building behaves differently, due to various factors such as climate, location, and building occupants; a specific digital twin is a better representation than a general model such as EnergyPlus. For example, some buildings may have natural ventilation, whereas others may have occupancy-based HVAC control; these variations in building resources require a different representation to ensure that the objectives to optimize are measured correctly. The digital twin representation is largely useful for three reasons.

1. 1. The system can save historical data for future analysis of historical patterns. Since the data is timestamped, the historical data could be read as a stream to reproduce the learning process of the optimizing model.

2. 2. The system can be used to provide real-time insights for building occupants and building managers. The saved data can visualize the current building state with the occupants’ locations, comfort level and environmental status such as air quality and energy.

3. 3. The digital twin’s output can be used as inputs for intelligent systems such as a recommender system. The digital twin is based on the collected real data so it can be used for data augmentation for the learning algorithm.

Because of these advantages, we create a digital twin for our deployments (Section 7), allowing us to save historical building states and leverage these states to perform data augmentation and alleviate the cold start problem (Section 6.3).

Our system has three optimization goals: energy, comfort, and air quality. Thus, in addition to monitoring the location of occupants, the digital twin should be able to accurately measure real-time energy consumption, occupant thermal comfort, and air quality. These are measured by the networked sensing layer (implementation discussed in Section 5).

To efficiently store and update the sensed energy, air quality, comfort, and location of occupants in a digital twin representation, we first model the building as a tripartite graph structure, as in Wei et al., Reference Wei, Chen, Vega, Xia, Chandrasekaran and Jiang2017a. This tripartite data structure separates the energy-consuming resources, physical spaces, and occupants into three object layers, where energy-consuming resources are connected to the spaces serviced, and occupants are connected to the spaces they occupy as shown in Figure 5. The objects in each layer are a one-to-one representation of elements in the physical deployment. This enables fast updates of individual object parameters when new data is received from the sensors, such as a fan coil unit power value or space temperature and humidity value. Additionally, as described by Wei et al. (Reference Wei, Chen, Vega, Xia, Chandrasekaran and Jiang2018a), updates to the graph can be quickly propagated to related elements in the graph, either immediately or on demand. As an example, a certain location may receive a new temperature value; this temperature value can be propagated to the connected occupant objects to update estimated thermal comfort immediately.

Figure 5. The digital twin is constructed using a tripartite graph structure, from Wei et al. (Reference Wei, Chen, Vega, Xia, Chandrasekaran and Jiang2017a) and Chen et al. (Reference Chen, Norford, Samuelson and Malkawi2018), to store relations between energy resources, spaces, and occupants. The building state can be extracted from the graph structure to build visualizations.

A real-time building state can be quickly extracted from the tripartite graph structure, which enables three important applications. First, different visualization tools can be built on the building state to alert building managers of potential improvements. The inclusion of personalized thermal comfort can provide additional information to inform of potential thermal comfort improvements. Second, the building state can be recorded in a database to build datasets for analyzing historical patterns and training prediction models. Finally, the real-time building state can be used as input to complex systems such as a recommender system. In this work, the digital twin is utilized for historical data to train simulation environment prediction models and to provide the building state as a real-time input to our recommender system.

6.2. Building modeling

Leveraging the digital twin, we can collect data on the current state of the building. From the building’s data streams, we collect the following features per room: HVAC energy consumption, lighting energy consumption, temperature, humidity, PM₁₀, PM_2.5, and room temperature setpoint. We also ascribe a room location to each building occupant, including a null indicator if the occupant is absent. These features encompass the building’s state in our modeling framework. Historical data on the building’s state can be used to train statistical models to predict future states for each of the rooms in the building. The key features that vary according to occupant actions, and therefore constitute our recommendations, are the room temperature setpoint and the occupants’ locations. These two features ultimately impact the building’s energy consumption as well as the occupants’ experiences of thermal comfort and air quality.

6.3. Data augmentation for exploring more building states and actions

From our digital twin, we develop a simulation environment that enables the creation of a large number of potential future states, which is used to evaluate recommendations and train our recommender system, as shown in Figure 6. The simulation environment leverages statistical models to simulate future building states based on occupant actions and the input building state, collected from our deployments and digital twin. We first discuss how we simulate and predict key modules: HVAC energy, lighting energy, indoor comfort, indoor air quality prediction, and comfort (Section 6.3.1); the reward is computed as standard in Equation (1) in Section 4.1. Next, we introduce how we incorporate this simulator into training our deep Q-network (Section 6.3.2).

Figure 6. To alleviate cold start, we create a simulation environment based on the digital twin, which takes as input the current building state and simulates the next state, energy savings, comfort improvement, and air quality improvement based on an action.

6.3.1. Predicting future states

1. HVAC energy: Predicting HVAC energy is challenging due to nonlinearities associated with HVAC control and outdoor environmental conditions. As discussed in Section 2, physics-based models such as EnergyPlus have limited ability to predict HVAC energy, especially when detailed design documentation is unavailable and granular temporal scales (i.e., hourly) are required, which both apply in our setting. Data-driven surrogate energy modeling is a promising framework for addressing these limitations. However, these models are typically applied at the building level rather than the room level. Here, we introduce data-driven surrogate HVAC energy modeling at the room level to support our simulation engine.

We identified three regression models that have been shown to be effective in energy prediction tasks: artificial neural networks, random forests, and gradient boosting (Sun et al., Reference Sun, Haghighat and Fung2020). In this task, the target is room-level HVAC energy consumption and the features are as follows (aggregated to the hourly level):

• • Number of occupants

• • Outdoor temperature (T ₀) and humidity

• • Heating Degree Days ( $ HDD=65-{T}_0 $ in °F)

• • Cooling Degree Days ( $ CDD={T}_0-65 $ in °F)

• • Temporal features: day of the year, day of the week, hour of the day

• • Historical HVAC energy data (48-h sliding window)

Energy prediction in this setting resembles time-series forecasting with exogenous features. Because historical energy data is available from the digital twin, we can also adapt the model to include historical HVAC data as a feature. For example, if we are only interested in one-step-ahead prediction, we can include all historical data up until the next time step as features.

We used a temporal 80–20% split to create training and testing sets. As a state-of-art comparison, we also built an EnergyPlus model based on historical building state data collected from our deployments and performed standard calibration measures, following the procedure in Miller et al. (Reference Miller, Thomas, Irigoyen, Hersberger, Nagy, Rossi and Schlueter2014). We used the coefficient of variation of the root mean squared error (CV(RMSE)) to compare the models:

(7) $$ CV(RMSE)=\frac{1}{{\overline{Y}}_i}\sqrt{\frac{\sum_{i=1}^N{\left({Y}_i-{\hat{Y}}_i\right)}^2}{N}} $$

where $ Y $ is true energy in the test set, $ \hat{Y} $ is predicted energy in the test set, and $ \overline{Y} $ is the average of the true energy, for N predictions. CV(RMSE) is commonly used to assess energy prediction performance in buildings, where a value less than 30% is considered a well-calibrated model (ASHRAE, 2002). We found that the random forest model produced the best results, with an average CV(RMSE) of 28.6% across the rooms, compared to EnergyPlus’s 94.3%. We therefore implemented the random forest model for 1-h-ahead energy prediction. A comparison between the random forest prediction and the EnergyPlus prediction for one of the rooms in our testbed is shown in Figure 7.

Figure 7. Comparison of EnergyPlus and random forest (RF) prediction for a single room in our deployments.

Through the data-driven surrogate modeling paradigm, we were able to build a simulation engine that is far more accurate than the standard tools used for HVAC energy simulation. In addition to the random forest model, we also incorporated an HVAC modifier based on manual thermostat control in the building. Each room in our study and deployments has a fine-grained manual control option, whereby occupants can make adjustments from neutral to “cool” or “warm.” We investigated the relationship between manual thermostat changes and HVAC energy and found that thermostat changes tend to increase the HVAC energy consumption of the room. Over the course of our study, manually applying the “cool” setting tends to increase energy consumption more than manually applying the “warm” setting. We also incorporated these modifiers into our overall simulation engine.

2. Lighting energy: Based on the data from our collected building state history, we assume that lighting operation can directly follow occupancy patterns: lights are on whenever a room has at least one occupant and off whenever a room is unoccupied since most modern office lights are controlled with motion sensors. We note that there are situations in which this direct control is not realized, for example when occupants override the system or when motion sensors experience errors. For our modeling purposes, we leveraged the data from the building’s digital twin to identify the “on” and “off” states for each room’s lighting fixtures. Based on the data, we were able to assign a specific power consumption for each occupied room in our building model.

3. Indoor temperature: We model the relationship between thermostat settings and indoor temperature from data collected from our networked sensing deployment. Individual rooms in the building include thermostats that give occupants the option to change the temperature. To investigate the empirical relationship between these settings and actual indoor temperature, we built linear regression models for each room that include an indicator variable for each of the possible thermostat settings at each timestep as features and the actual temperature at each timestep as the response variable. In our deployments, the thermostats had three possible settings, “warm,” “cool,” and “neutral”. We found that, on average, setting the thermostat to “warm” increases temperatures by 1.88°F and setting the thermostat to “cool” decreases temperatures by 1.80°F.

4. Air quality: Our air quality prediction task closely follows that of indoor temperature. We would expect a change in the thermostat setting to increase airflow in the room because the heating and cooling system would need to supply additional air to affect air temperature. We would also expect the additional air to be cleaner, due to the filters in the HVAC system. We would therefore expect such thermostat actions to decrease PM concentrations. However, these dynamics were not clearly evident in the data—the regression models did not produce significant relationships for most rooms. For each room, we did include small factors based on the direction of the relationship between HVAC energy and PM concentrations.

5. Comfort: In real deployments, we more accurately measure thermal comfort on the PMV ASHRAE 55 scale using thermal cameras and indoor temperature (Section 5). Because it is difficult to predict future temperatures at each pixel for each thermal camera, we instead simulate the comfort levels of each occupant by directly using PMV ASHRAE 55 and substituting indoor temperature and standard values, just like in Wei et al., Reference Wei, Liu, Kang, Yang and Jiang2021.

6.3.2. Training

To train and remedy the cold start problem, we create an offline training environment by integrating our simulator, with the deep Q-network in a tightly coupled control loop, which allows for rapid data generation and learning. An illustration of the training environment is shown in Figure 8.

Figure 8. For a building state, the reinforcement learning agent provides an action to the simulation environment. The next state, energy savings, comfort, and air quality improvements are returned to the agent to tune the policy.

Training runs in episodes, with fixed $ \Delta =1 $ h time steps. In each episode, the simulation environment is instantiated with occupant locations and building states based on historical data. The state s is provided to the deep Q-network, which outputs predicted changes in energy, comfort, and air quality for each possible action from this state. An action is chosen based on softmax selection (Equation 6), just like how recommendations are displayed.

The action is sent to the simulation environment (Section 6.3) and performed in simulation, and the episode advances to the next step, producing a new building state (next state $ {s}^{\prime } $ ) to be sent to the deep Q-network. Expected rewards for energy, comfort, and air quality are estimated and stored with the state in a pool of samples for experience replay.

6.4. Adapting to user preferences

It is not productive to suggest actions that an occupant has consistently rejected. To address this, we estimate the probability that an occupant will accept a recommendation by counting the recommendations that occupants have rejected (“feedback” as described in Section 4.3.2). We add a penalty to the ranker scoring function (Equation 6) that penalizes recommendations with more rejections.

Second, a person’s preferences may change over time. As such, we allow building managers to begin retraining the recommender system after an interval of time. Once started, RECA will automatically augment the data observed and stored in history using our simulation environment (Section 6.3) to retrain the deep Q-network.

6.5. Conflicting actions

In real deployments, recommender systems serve multiple occupants concurrently with real-time recommendations. This poses two problems for a recommender system. First, different occupants may choose conflicting recommendations, which may cause suboptimal or even negative results. For example, one occupant may choose to increase the setpoint temperature to reduce energy, while a second occupant in the same location may choose to decrease the setpoint temperature to increase comfort. Second, once an occupant has chosen a type of recommendation, there should be a period of time before the occupant is shown that type of recommendation again. Receiving a move recommendation soon after selecting a different move recommendation can be a poor user experience, and can potentially counteract objective improvements.

To address these challenges, we temporarily remove any move or setpoint recommendations for an occupant at a location once a move or setpoint recommendation has been accepted, for a period $ {t}_s=1 $ h. Additionally, when a setpoint recommendation has been selected by an occupant for a location, no other occupants will be given setpoint recommendations for the remainder of t_s.

7.1. Evaluation using simulation environment

As the simulation environment is responsible for augmenting data for training and retraining the deep Q-networks, it is critical to evaluate the performance of the models in simulation before deploying them in the real world. We tested our recommender system with different objective weights (Equation 1) using simulated building episodes based on the building environments in our actual deployments, described in Section 7.2.

Evaluating these models in simulation involves simulating two states: one state representing that an action was performed, and the other state representing no action. We compare the energy consumption, thermal comfort, and air quality of occupants between the states. We calculate energy savings $ {E}_R $ as the difference in energy consumption between the two states over the nth timestep of length $ \Delta $ , where $ {P}_d\left(\cdot \right) $ and $ {\hat{P}}_d\left(\cdot \right) $ denotes the power of resource d in the baseline state and the state with action (Equation 8).

(8) $$ {E}_R^n={\int}_{t=n\Delta}^{\left(n+1\right)\Delta}\sum \limits_d\left({P}_d(t)-{\hat{P}}_d(t)\right) dt. $$

We calculate thermal comfort and air quality improvement by taking the average PMV $ \left( PMV\left(\cdot \right)\right) $ and air quality ratings ( $ AQ\left(\cdot \right) $ ) for each occupant $ o\in O $ in room R_o (Equations 9 and 10).

(9) $$ {C}_R^n=\frac{\sum_{o\in O}{\int}_{t=n\Delta}^{\left(n+1\right)\Delta}\hat{PMV}\left({R}_o,t\right)- PMV\left({R}_o,t\right)\hskip0.1em dt}{\Delta \cdot \mid O\mid } $$

(10) $$ {Q}_R^n=\frac{\sum_{o\in O}{\int}_{t=n\Delta}^{\left(n+1\right)\Delta}\hat{AQ}\left({R}_o,t\right)- AQ\hskip0.1em dt}{\Delta \cdot \mid O\mid } $$

We simulated the performance of different models on 10 000 episodes with semi-randomized occupant start locations. As shown in Table 1, we simulated episodes using the deep Q-network architecture with and without the embedding layer, and set weights, in Equation (1), for four different emphases: energy $ \left(\alpha >>\beta =\gamma \right), $ comfort $ \left(\beta >>\alpha =\gamma \right), $ air quality $ \left(\gamma >>\alpha =\beta \right) $ , and joint optimization $ \left(\alpha =\beta =\gamma \right). $ We note that the architecture without the embedding layer performed worse for each emphasis in comparison to the architecture with the embedding layer and would often recommend actions that negatively impact the objectives. For example, in the comfort emphasis, RECA with embedding improved average PMV by 0.31 on a 3-point scale, compared to 0.03 without the embedding. Although the energy savings achieved for the comfort emphasis without embedding (22.2 kWh) exceeded the savings with the embedding (14.9 kWh), the system was tuned to emphasize and achieve comfort improvements at the expense of other objectives. Similar trends can be observed for all other emphases, while the joint emphasis achieves a more balanced improvement across all three objectives because it weights all three similarly in importance.

Table 1. Comparison of our deep Q-network architecture with and without an embedding layer, against existing strategies, on simulated building episodes with four different weighting combinations to emphasize different optimizations

Additionally, the service strategy can only improve energy savings because it only turns down services in areas with no occupants. Our system also outperforms the setpoint only strategy across all emphases because our system can not only change setpoints but also recommend more complex action sequences, such as grouping occupants with similar temperature preferences together.

7.2. Recommender system study

We deployed RECA in two commercial buildings over four weeks to evaluate energy savings, comfort improvement, and air quality improvement of different strategies and recommendation policies. In these deployments, RECA ran on $ \Delta =1 $ h time steps. Building A is an office building consisting of 10 rooms, where seven are rooms of cubicles, three are lab areas, and one break room. Building B is another office building consisting of eight rooms, where four are open work areas, two are closed office spaces, and two are shared spaces. We recruited 10 occupants in Building $ A $ and 13 occupants in Building B, ranging in age from 22 to 40 from various academic disciplines, and collected recommendation feedback from participants over the course of four weeks. We obtained approval from Columbia University’s internal review board for all our deployments.

Since our deployment can only measure one building state sequence, we decided to use the sensed building state to measure the performance of RECA, while we simulate the energy consumption, comfort, and air quality as the baseline, using the tools from our simulation environment (Section 6.3). This is because once a user takes a recommendation, we can no longer observe the state of the building in the scenario where the occupant does not perform the action, which is required to measure improvements in energy consumption, comfort, and air quality. As such, we simulate scenarios where users do not accept any of the recommendations as the baseline. As opposed to the pure simulation environment, we directly measure changes in energy consumption, comfort, and air quality in our real deployments as a result of RECA.

Figure 9 shows the number of accepted recommendations, broken down by recommendation type (move and setpoint). We scheduled RECA to retrain itself at two weeks (Section 6.4), using data from the first two weeks. The control period refers to the first two weeks, while the adapted period refers to the latter two weeks after retraining. There is a dramatic increase in recommendation acceptances, across both types of recommendations, in the adapted period because the retraining process allowed RECA to learn recommendations that users are more likely to accept. Across both deployments, we saw an 80% increase in accepted recommendations during the adapted period.

Figure 9. Acceptance rate during the first two weeks (control) and second two weeks after retraining (adapted) for the setpoint (left) and move (right) recommendations.

7.3. Learned action sequences

Throughout our study, we noticed and categorized three types of regularly occurring action sequences, that RECA recommended, with greater effects than individual actions: location optimization, group consolidation, and group disbanding.

Location optimization: The most common complex action sequence that we observed is location optimization. This sequence is typically made up of a setpoint change and a move recommendation. The main purpose of this sequence is, in most cases, to reduce the energy consumption of the starting location, without incurring a thermal comfort or air quality penalty by moving the occupant to another location. An example of this action sequence is shown in Figure 10. Initially, a single occupant is working in location A. At a certain time, indicated by the red line, the occupant increases the setpoint temperature of location A and moves to location B. The normal consequence of increasing the setpoint is a reduction in thermal comfort for the occupant as the temperature gradually increases; however, moving to location B leads instead to an increase in thermal comfort due to the environment in location B being closer to the occupant’s thermal preference, as shown in the bottom left plot. The green highlights the improvement in the occupant’s measured PMV after the occupant moves to location B (lower value = more comfortable). Increasing the setpoint temperature in location A (upper left) reduces the energy consumption of the space, as shown by the green. After the occupant moves to location B, the lights and electricity needed to service location B increases, as shown by the red (upper right). However, this increase is more than offset by the savings in location A. The air quality experienced by the occupant remains relatively stable (lower right).

Figure 10. Location optimization: At the red line, the occupant moves to location B, and HVAC service is reduced in location A. Due to differences in environment, the occupant’s thermal comfort and air quality are improved.

Group consolidation: Action sequences involving multiple occupants can enable even more optimization opportunities. In group consolidation, several occupants are brought to the same location often with a reduction in energy consumption in the start locations. As illustrated in Figure 11, occupants 1 and 2 are in locations A and B, respectively. At different times, denoted by the blue and red lines, occupants 1 and 2 increase the setpoint temperatures of locations A and B, and subsequently move to location C. By increasing setpoint temperatures, locations A and B both experience a decrease in energy consumption as highlighted in green in the upper plots. The thermal comfort and air quality for both occupants change depending on the environmental differences at locations A–C. In this example, the temperature preferences of location C are more suited for occupant 1 than occupant 2. As such, we see that the thermal comfort of occupant 1 improves (middle left plot highlighted in green), while occupant 2 decreases (middle right plot highlighted in red). The air quality for both occupants is improved in location C, as compared to locations A and B (lower plots highlighted in green). Note that in location optimization and group consolidation, the location that the occupants move to is critical. For example, if the destination location is too hot or too cold, the result of the action sequences may lead to an overall decrease in thermal comfort.

Figure 11. Group consolidation: At the blue and red lines, occupants 1 and 2 move to location C (not shown). Locations A and B reduce HVAC and lighting service, leading to energy savings. Comfort and air quality for both occupants change due to environmental differences.

Group disbanding: The final category of action sequence that we observed is group disbanding. The primary challenge in optimizing thermal comfort specifically, is that each occupant has a different thermal preference. Two occupants with different thermal preferences in the same location will require a compromise in setpoint temperature to prevent significant discomfort from one or both of the occupants. Group disbanding seeks to resolve this challenge by separating the occupants into different locations and group occupants with similar thermal preferences. In Figure 12, occupants 1 and 2 are in location A and have different thermal preferences. At the red line, occupant 2 is recommended to move to location B. The thermal comfort and air quality of occupant 2 increase as a result of the change in location (middle and lower right plots highlighted in green). Note that after occupant 2 leaves, occupant 1 changes the setpoint temperature higher at the blue line, leading to reduced energy consumption (upper left highlighted in green), improved thermal comfort (middle left highlighted in green), with a slight decrease in air quality (lower left highlighted in red).

Figure 12. Group Disbanding: At the red line, occupant 2 moves from location A to location B. Since only occupant 1 remains, HVAC service is reduced in location A at the blue line, leading to a comfort improvement for occupant 1.

7.4. Joint optimization results

In this section, we show the strengths of RECA against existing solutions in two real commercial buildings. Specifically, we show the versatility of RECA, allowing building managers to configure RECA to improve any combination of energy, comfort, and air quality. We also demonstrate how incorporating humans-in-the-loop allows RECA to further improve these objectives over existing solutions that only reduce services and change temperature setpoints.

Case 1: Energy optimization: One of the most studied problems in commercial buildings is energy optimization. To study the performance of RECA on energy optimization, we select weights of the ranker to emphasize energy savings ( $ \alpha >>\beta $ and $ \gamma $ ).

Figure 13—top shows the percentage improvement of energy consumption, occupant thermal comfort, and occupant air quality. Since the baseline strategy only reduces service in locations with no occupants, only energy consumption is improved. In comparison, the setpoint-only strategy further improves energy consumption but reduces occupant comfort. The reason is that increasing setpoint temperature reduces HVAC service, but also increases the temperature, leading to a decrease in occupant comfort. Finally, by including move recommendations, the system achieves a further 8% and 6% increase in energy savings without sacrificing comfort in buildings A and B, respectively. As described in Section 7.3, moving occupants enables locations with high energy requirements to reduce service without incurring penalties to thermal comfort. However, we also observe a decrease in air quality in building A. There are two reasons for this. First, moving more people into the same room concentrates emissions; our system also reduces HVAC services in more rooms, which leads to a decrease in air quality because less air is being filtered. Second, in this scenario, the building manager configured our system to increase energy efficiency without considering other factors. As such, our system focused on reducing as much energy as possible, even at the cost of other factors.

Figure 13. Energy savings, comfort and air quality improvements, emphasizing energy savings (top), balanced improvements (middle), and comfort, and air quality (bottom) in two deployments (A and B).

Case 2: Joint co-optimization: In some cases, building managers may wish to save energy without sacrificing comfort or air quality. Joint co-optimization is a more complex problem, as many actions lead to tradeoffs between the three objectives. To evaluate the potential of our recommender system to jointly optimize all three objectives at once, we set the weights of the ranker to balance the three objectives ( $ \alpha =\beta =\gamma $ ). We also deployed the setpoint and service strategy, which combines both the setpoint only—comfort strategy with the service strategy, as a comparison.

As shown in Figure 13—middle, the setpoint and service strategy improves thermal comfort and energy consumption, while minimally impacting air quality. However, the complete recommender system makes additional improvements of 5%, 7%, and 6%, for building A, and 9%, 9%, and 8%, for building B, in energy consumption, thermal comfort, and air quality. With move recommendations enabled, the recommender system can utilize more complex action sequences to find significant optimization opportunities that are not possible in the setpoint-only strategy. Compared with case one, where a building manager may only be concerned about energy savings, we see that our system can jointly improve all three aspects at the same time by incorporating humans in the loop.

Case 3: Comfort and air quality co-optimization: In certain cases, it may be desirable to allow increases in energy consumption to significantly increase occupant comfort and air quality. Due to the recent COVID-19 pandemic, improving air quality is a priority in work environments, while improving occupant thermal comfort can improve productivity. We deployed our recommender system with higher comfort and air quality weights ( $ \beta =\gamma >>\alpha $ ) to encourage recommendations of actions with high thermal comfort and air quality improvements. In comparison, we deployed the setpoint only—with air quality emphasis strategy for A/B testing.

As shown in Figure 13—bottom, the setpoint only baseline is able to prioritize comfort and air quality improvements with the tradeoff of increased energy consumption. However, by treating occupants as immovable objects, we observed that many of the setpoint temperature changes were compromises for multiple occupants. In contrast, the recommender system that utilizes move recommendations shows a dramatic increase in comfort and air quality improvements of 21% and 5% for building A and 11% and 4% for building B. We noticed multiple instances of group disbanding, which allows for more personalized thermal comfort by separating groups of people with different thermal comfort preferences.

7.5. Scalability

The size of the network scales $ O\left(|O|\times |S|\right) $ , where $ \mid O\mid $ is the number of occupants we engage and $ \mid S\mid $ is the number of locations in the building. Although this is linear in both occupants and locations, it can become expensive if the building gets large. Table 2 shows the execution time and scalability of RECA as the number of occupants and spaces increases. We see that the majority of the computation comes from the DQN. However, even as the number of occupants and spaces increase to over 800 and 400, respectively, the total execution time is around 15 s. This latency is more than acceptable for our system, which updates once every hour.

Table 2. Execution time for each component of RECA as the number of people and spaces increase

In even larger deployments where computation time may exceed one time step, we can use the observation that people will typically use only a small portion of the building. As such, we can reduce the state-action space by eliminating infeasible actions and/or creating multi-agent systems with smaller models to manage portions of the building. We plan to explore these avenues in future work.