Impact Statement

In this study, we show how machine learning (ML) can be used to predict a building’s occupancy type. We identify several features from open datasets capable of predicting building occupancy type while also evaluating multiple ML approaches. Residential buildings significantly outnumber commercial buildings, with commercial buildings significantly outnumbering other types of buildings. We evaluate multiple strategies for handling this class imbalance and show how Bayesian neural networks can assist in uncertainty quantification. Our results suggest strategies for utilizing ML-based approaches across the continental United States.

2. Related work and the need for ML

OpenStreetMapFootnote ³ (OSM, OpenStreetMap, 2022) is a crowd-sourced initiative that aims to provide free and open access to spatial data on a global scale. OSM’s representations of streets, natural landmarks, and building outlines are often more comprehensive and accurate than traditional data sources such as the CIA World Factbook and United States Census Topologically Integrated Geographic Encoding and Referencing (TIGER)/Line data (Fan et al., Reference Fan, Zipf, Fu and Neis2014; Boeing, Reference Boeing2017; Jacobs and Mitchell, Reference Jacobs and Mitchell2020; Moradi et al., Reference Moradi, Roche and Mostafavi2022). Yet, even in areas with abundant data, the free-form and optional nature of OSM’s building occupancy type attribute results in most mapped buildings having no reliable occupancy type. The lack of reliable building occupancy type has led several researchers to explore using machine learning to predict this value. For instance, (Atwal et al., Reference Atwal, Anderson and Pfoser2022) achieved 98% accuracy in the binary classification task of predicting residential versus non-residential occupancy type. To achieve this, the authors used other features within the OSM dataset to train a decision tree. While the accuracy result is impressive, one of the required inputs to the decision tree is the so-called OSM building tag. There are two major challenges with this requirement. First, OSM building tags are rarely available and when they are they can be ambiguous. As noted by (Hoffmann et al., Reference Hoffmann, Wang, Werner, Kang and Zhu2019) “It is important to note that, apart from the vocabulary difference and spelling error in the building tag, OSM also faces ambiguities in their finer classification scheme that is defined in the OSM Wiki.” Such a decision tree could not be applied to tens of millions of buildings when used at CONUS scale. Second, having access to a highly informative tag such as “house” or “apartment” would seem to negate the need for ML.

Hoffmann et al. (Hoffmann et al., Reference Hoffmann, Wang, Werner, Kang and Zhu2019) used a deep learning approach that predicted building type from a combination of aerial and street view images. They designed a multiclass prediction system in which the available categories were: commercial, residential, public, and industrial. (Hoffmann et al., Reference Hoffmann, Wang, Werner, Kang and Zhu2019) concludes that with a multiclass setup and the challenges of OSM, a classification accuracy of about 60–80% on average would be a realistic expectation.

Our goals in this research are fourfold. First, to demonstrate the need for both binary and multiclass classification. Second, to see if the classification accuracy laid out by (Hoffmann et al., Reference Hoffmann, Wang, Werner, Kang and Zhu2019) can be achieved using publicly available data and without the need for proprietary APIs and commercial image collections. Third, to determine which building-related features openly available at CONUS scale have the most predictive power. And, fourth, to provide a method of capturing uncertainty in predictions. If the challenges of open datasets like OSM coupled with large class imbalances limit prediction accuracy, then we need methods for determining when ML algorithms are making reliable predictions.

It is worth noting that our building occupancy type classification is fundamentally different from the similarly sounding building occupancy classification. Numerous studies in the literature (e.g., (Kanthila et al., Reference Kanthila, Boodi, Beddiar, Amirat and Benbouzid2022; Sayed et al., Reference Sayed, Himeur and Bensaali2022)) are focused on the occupants’ presence, power usage habits, and interior conditions with the goal of enhancing building operation and management. Their focus is on curtailing energy consumption and reducing costs. Our work focuses on determining building type (i.e. residential, commercial, industrial, educational, etc.). We are interested in the type of structure and not how its occupants are utilizing power, heating, and cooling. This difference in application results in a need for different types of data, which results in different machine-learning needs and challenges.

3. ML dataset

No CONUS-scale ML-ready dataset for building occupancy type currently exists. It is not even apparent which building-level features available openly at the CONUS scale have the most predictive power. We do not claim that the features utilized in this study are the most predictive. Rather, we gather a sampling of different types of open data available at the building level and perform exploratory analysis. Our goal is to predict a building’s type (e.g., residential/commercial/industrial) based on known features of the building (area, location, etc.). To accomplish this, we gather a number of building features from open datasets. Our sources of building features are the Open Street Map (OSM), the Multi-Resolution Land Characteristics Consortium (MRLC),Footnote ⁴ the United States Census’ County Business Patterns (CBP),Footnote ⁵ and the United States Census’ American Community Survey (ACS)Footnote ⁶. We have spatially aligned all features by using OSM longitude and latitude to determine which county the building resides in. We have then gone to the ACS and looked up socio-economic county data such as median income and housing density. Imperviousness values are taken from the MRLC. The building area polygons from OSM are overlayed with 30-meter resolution imperviousness data and a weighted average of the imperviousness underneath the building footprint was computed. The resulting ML-ready data have the features listed in Table 1.

Table 1. Features utilized in our machine learning training data

Assessing the accuracy of building occupancy-type predictions involves a source of “ground-truth” data. We utilize the US structuresFootnote ⁷ dataset maintained by FEMA and created in conjunction with DHS Science and Technology and Oak Ridge National Laboratory. US structures was created by extracting building outlines via commercially available satellite imagery. Building occupancy type (e.g., residential, commercial, industrial) was then determined from a combination of local governments who agreed to share it, open data from the National Geospatial-Intelligence Agency (NGA), Census housing data, and parcel data. These US Structures building types are what we are trying to predict. The US Structures ‘OccupancyType’ is the final column in our training data and our source of “ground-truth” values. We note that the US Structures dataset is smaller than OSM and currently only available in Alabama, Florida, Georgia, Louisiana, Mississippi, North Carolina, South Carolina, Texas, and Virginia. Thus, while a good source of ground-truth data, US Structures does not preclude the need for machine learning. Utilizing North Carolina as a test site, we have created an ML-ready dataset of 1,078,144 buildings with known building occupancy types. We use Alabama US Structures as a test of transferability. An ML model trained in North Carolina is applied to Alabama to asses if CONUS-scale prediction can be done with a single ML model or requires state-specific models. US structures recognizes eight building occupancy types and the occurrence of these types in North Carolina is shown in Table 2.

Table 2. Summary statistics of building occupancy types in North Carolina

Evident from Table 2 is that we have a significant class imbalance. The residential buildings, for example, outnumber the commercial buildings by 15 to 1. The residential buildings outnumber the agricultural buildings by nearly 600 to 1. This can be a challenge for machine learning classification algorithms. The algorithm may not “learn,” but rather obtain high accuracy simply by picking the majority class. We look at techniques for dealing with such imbalance.

4. Machine learning algorithms and evaluation metrics

Applications using building-type information demand multiple types of classification. For example, during emergency response, first responders may only be interested in identifying residential buildings. In this use case, binary classification (residential vs. non-residential) is sufficient. However, for economic forecasting applications, a finer-grained distinction is needed (i.e., residential vs. commercial vs. industrial). In such a scenario, multiclass classification is needed to determine a comprehensive economic assessment. We provide examples of both scenarios. Specifically, we design a set of supervised learning models in which building features (location, square footage, proximity to other resources, etc.) are used to predict building occupancy type. We assess the accuracy, precision, and recall of two machine learning algorithms while also investigating techniques for dealing with unbalanced classes.

For the binary classification scenario, all non-residential building types are changed to “Other.” In the multiclass scenario, we attempted to accurately classify all eight building occupancy types recognized by US structures. As detailed in subsequent sections, the accuracy was too low to be practically useful. We settled on a three-class model of Residential, Commercial, or “Other.”

In both binary and multiclass scenarios, the OsmNearestRoad and OccupancyType, which are initially text, are encoded to numbers and we scale the data (for neural networks) based on standard scores. Our ML dataset is split into training and testing portions using the standard 80/20 training/testing split. In other words, 80% of the buildings in North Carolina are used to train the various ML algorithms and the remaining 20% is held aside to evaluate the trained algorithms. The training data are applied to multiple variations of random forest and neural network approaches. We include the standard random forest approach as a baseline and then compare variations designed to better handle class imbalance.

Trained ML models are then evaluated on the test set with the following metrics, which are based on the number of true positive (TP), false negative (FN), and false positive (FP) predictions.

Balanced accuracy—a variation of accuracy used in classification problems with imbalanced datasets defined as the average of recall obtained on each class. Recall of each class is found from

$$ \mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} $$

We also report precision, which is defined as

$$ \mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}} $$

The F1 score is the harmonic mean of precision and recall given by

$$ \mathrm{F}1=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}} $$

Precision, recall, and F1 scores exist for each class. In the binary classification scenario, for example, there are precision, recall, and F1 values for Residential predictions as well as precision, recall, and F1 values for non-residential Predictions. We provide average precision, recall, and F1 values when summarizing our results in tables. The reported averages are the so-called macro average, which is computed by taking the arithmetic mean (unweighted) across the classes. This approach treats each class with equal significance.

4.1. Binary classification

Table 3 shows the distribution of building occupancy types after non-residential buildings are converted to “Other.” There still exists a significant imbalance in the building occupancy types. We trained a random forest model using 5-fold cross-validation to ensure accuracy was not dependent on how the data was split.

Table 3. Building occupancy type distribution for binary classification

Undersampling refers to a group of techniques designed to balance the class distribution. Undersampling techniques remove examples from the training dataset that belong to the majority class in order to better balance the class distribution. This is in contrast to oversampling, which involves adding examples to the minority class in an effort to reduce the class distribution imbalance. Near Miss (Zhang and Mani, Reference Zhang and Mani2003) refers to a collection of undersampling methods that select examples based on the distance of majority class examples to minority class examples. Distance is determined in feature space using Euclidean distance. NearMiss allows us to keep majority class examples that are on the decision boundary leaving 101,453 residential buildings and the same number of non-residential buildings.

Perhaps the simplest approach to handle a severe class imbalance is to change the decision threshold. A random forest model is comprised of multiple decision trees. We can obtain from the model the proportion of decision trees that voted “residential” and the proportion of decision trees that voted “non-residential” for a given building. An obvious default approach is to set the threshold at 0.5. If the proportion of decision trees voting “residential” is above 0.5 then the algorithm predicts “residential.” Otherwise, the algorithm predicts “non-residential.” Threshold moving simply moves this threshold attempting to achieve higher accuracy. For example, we may require a proportion of 0.6 or greater to predict “residential.” Threshold moving can be effective for class imbalance because it helps avoid simply predicting the majority class. In our case, more evidence is needed before predicting “residential.”

The “best” threshold to use is determined by finding the threshold with the maximum F1 score, a metric which is the harmonic mean of precision and recall. Figure 1 illustrates the results of threshold moving applied to our binary classification random forest model. The points connected by a solid line show the change in precision and recall as the threshold is modified. The large solid black dot highlights the location where precision and recall combine to maximize F1. Our experiments revealed an optimal threshold of 0.55.

Figure 1. Precision recall curve for random forest threshold moving.

We also trained a binary classification neural network to compare to the random forest approaches. Because of the class imbalance, we want the neural network to pay more attention to the fewer examples of non-residential buildings. A common technique for achieving this is to weight the classes using:

$ \mathrm{weight}=\left(1/{n}_{\mathrm{c}}\right)\times \left(N/2\right) $ (4.1)

where n _c is the number of buildings in the class and N is the total number of buildings.

We arrive at a residential weighting of 0.55 and a non-residential weighting of 5.31. The neural network itself has two hidden layers of size 30 and 15, respectively. The network was set to train for 150 epochs. However, early stopping was applied with the training set to stop if accuracy did not improve for seven consecutive epochs. After 36 epochs, training stopped due to no improvement in accuracy.

6. Bayesian neural networks for prediction uncertainty

The traditional approach to machine learning is to optimize a loss function to obtain an optimal setting of the model parameters. Numerous iterations over the training data allow the model parameters to be fine-tuned. The machine learning training process results in a set of optimal model parameters that are then used in subsequent predictions.

A Bayesian approach, by contrast, does not seek fixed model parameters. In a Bayesian neural network (Wilson and Izmailov, Reference Wilson and Izmailov2020) the network weights are distributions. Passing the same input through the trained network will result in different predictions as different weights are chosen from the distributions on each pass. Getting a distribution of predictions for each input is useful as the ensemble of predictions can be used to quantify uncertainty in neural network weights—commonly referred to as epistemic uncertainty—and the probability of being in each class leads to a determination of prediction uncertainty—commonly referred to as aleatoric uncertainty.

Our Bayesian neural network is implemented using the TensorFlow Probability software library (Dillon et al., Reference Dillon, Langmore, Tran, Brevdo, Vasudevan, Moore, Patton, Alemi, Hoffman and Saurous2017). The other layers in our neural networks are standard Dense layers from the Keras library (Chollet and others, Reference Chollet2015). Aleatoric uncertainty is quantified using Shannon entropy, which is common for classification problems (Hüllermeier and Waegeman, Reference Hüllermeier and Waegeman2021). Shannon entropy is defined as

$$ H=-\sum \limits_{i=1}^n{p}_i\times \mathit{\ln}\left({p}_i\right) $$

where n is the number of possible output categories (e.g., residential, commercial, and other) and $ {p}_i $ is the predicted likelihood of being in each of those categories. Entropy in information theory is analogous to entropy in statistical thermodynamics. Larger values of H correspond to the probabilities being spread more evenly across the possible classes. Thus, larger values of H correspond to the neural network being more uncertain in its prediction.

We create a Bayesian neural network for the binary classification problem and a second Bayesian neural network for the 3-class (residential, commercial, and other) problem. After training the networks, each building in the test set was passed through the network 100 times. The mean of the 100 predictions was computed and Shannon entropy was used to determine prediction uncertainty. This approach enables a user-defined uncertainty threshold. Predictions with uncertainty above the threshold are ignored due to the neural network having little confidence in the prediction. In this manner, the user is trading off number of predictions for higher prediction accuracy. Low values of the uncertain threshold—implying the neural network must have high confidence in the prediction—leads to fewer predictions being made; however, those predictions are of higher accuracy.

Figure 5 compares confusion matrices for the multiclass Bayesian neural network. The left panel shows the results when there is no uncertainty threshold, that is the neural network is forced to make a prediction for every building. The right panel shows the results for an example uncertainty threshold of 0.4 We are making about 50% fewer predictions in the right panel; yet, the neural network is much more confident—and correct—in those predictions.

Figure 5. Confusion matrices from the multiclass Bayesian Neural Network. On the left, results when all predictions are kept. On the right, results when prediction uncertainty is below 0.4.

7. Application and transferability

As a simple demonstration of how building occupancy type prediction can be used, we identified 219,054 buildings in the OSM North Carolina dataset not used in our train/test data that had unknown building occupancy types. Further, we obtained the 100-year flood map from the FEMA National Flood Hazards Layer. The phrase “100-year flood” is used to describe the extent of a flood that statistically has a 1-percent chance of occurring in any given year (Maidment, Reference Maidment2009). Here, it is used for illustrative purposes of a disaster preparedness application. Flood maps indicated that 5838 buildings out of the 219,054 unknown North Carolina buildings would be impacted by a 100-year flood. Our model predicted these to be 5452 residential, 349 commercial, and 37 “other.” The spatial distribution of the predicted residential and commercial buildings is shown in Figures 6 and 7, respectively. This simple example highlights two important points. First, despite open datasets such as OSM and US structures, there still exist hundreds of thousands of buildings in North Carolina with unknown occupancy types. Public safety assessments and economic forecasting of a natural disaster, such as a 100-year flood, are limited in their ability to fully account for the entire state. A machine learning approach, such as the one described here, can quickly and reliably predict those missing occupancy types.

Figure 6. Predicted residential flooding for buildings with previously unknown occupancy type.

Figure 7. Predicted commercial flooding for buildings with previously unknown occupancy type.

We obtained US structures data from Alabama to evaluate the transferability of our models. As mentioned previously, US structures has ground truth building occupancy types for nine states. Our goal was to assess if an ML model trained in one state could be used in another state. Unfortunately, this was not the case. The Alabama dataset contained 2.2 million buildings with known occupancy types. The multiclass model trained on North Carolina data was only able to accurately classify 48% of Alabama buildings by default. Accuracy increased using the Bayesian approach and uncertainty thresholds. However, the improved accuracy is not significant enough to think that one neural network model could be applicable across the continental United States as the distributions of the selected features vary too much between states. Put another way, while the selected features are useful for predicting building occupancy type, the weighting of each feature varies from state to state. Practitioners may need to train multiple state-specific neural networks. Alternatively, it may be feasible to train regional neural network models; yet, it is unclear to us at present how to optimally construct such training datasets. More research is needed into multi-state neural networks and the specific impacts of geographic features (that is imperviousness) and socio-economic features (that is median income and median area). We leave this for subsequent research.