2.1. Framework: product model

This framework is intended to be used during product development when the design is still somewhat flexible, but enough must be known about the product’s design to adequately estimate its performance. Items such as the performance, geometry, material and potential distribution model may be incorporated into the product model. Essential product attributes that may influence the product impact should be chosen as inputs for the ABM. For example, a designer might choose efficiency, comfort and ease of use as three potential design parameters to examine, explore and iterate. These attributes will also be used in the DCA as a part of an individual agent’s choice of adoption. Adoption modelling using DCA will be covered in depth in Section 2.5. The essential part of the product model is to find a relationship between product parameters and how they will influence both the adoption of the product and the metrics for measuring impact.

2.2. Framework: society model

The first step in building a society model is selecting a target population for the simulation. There are two types of information needed about a target population: survey data on user attitudes, behaviours and preferences, and demographic data to build a synthetic population in the model (Wassenaar et al. Reference Wassenaar, Chen, Cheng and Sudjianto2004).

Data on user preferences can effectively be collected in a survey. This survey includes how important the selected product parameters are to a potential user (Alberini Reference Alberini1995), as well as information related to the TPB categories of attitude, social norms and perceived behavioural control surrounding the adoption of the product (Ajzen Reference Ajzen1991). This framework focuses on a binary choice model, where the choice is to adopt or not adopt the product. A multinomial choice model may be considered where a user will choose between competing products, but this leads to a more complex model that is outside the scope of this paper due to needing preference data about competing products. Data about user preferences are needed for creating the adoption probability equations that will be discussed in Section 2.5.

The other set of data are about the demographics and behaviour of the target population. These data are used to inform agent actions and interactions. When complete knowledge of a population is not available, a synthetic population may be used. Synthetic populations create demographic data sets for agents that match the distribution of the population of interest (Müller & Axhausen Reference Müller and Axhausen2010). Synthetic populations allow for more accurate simulation of both the individual behaviour of agents, and interactions between agents that are affected by individual-level demographics (Gargiulo et al. Reference Gargiulo, Ternes, Huet and Deffuant2010). This helps ABM simulate emergent behaviour due to societal behaviour (Xu et al. Reference Xu, Glass, Lau, Geard, Graves and Clements2017).

Another key structure of the society model is the social network structure of the particular target population (Chen et al. Reference Chen, Heydari, Maier and Panchal2018). The framework in this paper uses a small-world network topology (Watts & Strogatz Reference Watts and Strogatz1998) that has individuals within the network connect to k closest neighbours, and each individual has a probability of rewiring a connection to that of another random individual from the entire population. Small-world networks have been used widely in ABM adoption and diffusion models (Cowan & Jonard Reference Cowan and Jonard2004; Schwarz & Ernst Reference Schwarz and Ernst2012; Rai & Robinson Reference Rai and Robinson2015) because they have been shown to follow patterns found in real-world social networks (Latora & Marchiori Reference Latora and Marchiori2001).

2.3. Framework: scenario model

The scenario model used in this paper defines the scope of the ABM. This includes model parameters such as the duration of the simulation, physical attributes of the world created for agents and initial conditions of inputs for the ABM (Macal & North Reference Macal and North2005). These initial conditions may include how many people initially have a product, the time period a simulation starts within, or an external force such as a law influencing an agents’ choice to adopt. Data gathered for the society model help generate building the physical attributes of the scenario with items such as the number of households or workplaces that may need to be created in the modelled world. It should be decided if one or multiple scenarios will be examined as design teams may be interested in the product impact over different contexts.

2.4. Framework: impact model

Much of the literature on assessing the social impact of products that are on the market will also apply to estimating impact. As outlined in SIA and Social Life Cycle Assessment, categories of impact need to be chosen and then indicators selected to measure the impact categories (Benoît et al. Reference Benoît, Norris, Valdivia, Ciroth, Moberg, Bos, Prakash, Ugaya and Beck2010; Fontes et al. Reference Fontes, Tarne, Traverso and Bernstein2018). Work on predictive social impact modelling adds the importance of determining how product parameters link to impact indicators (Stevenson et al. Reference Stevenson, Mattson and Dahlin2020). The framework in this paper utilises the 11 categories from Rainock et al. (Reference Rainock, Everett, Pack, Dahlin and Mattson2018) for social impact category selection. Clearly, the number of resources required to collect data will increase with the number of selected impact categories and indicators. Therefore, it may be important for the design team to narrow down the focus of the model to the number of impact categories that will enable data collection with available resources. Data from users should inform the selection of impact categories (Fontes et al. Reference Fontes, Tarne, Traverso and Bernstein2018).

2.5. Framework: ABM

Inputs from the product, society, scenario and impact models are integrated into an ABM. These inputs define the world that agents exist within, and the agents themselves. One of the key characteristics of any ABM is defining how agents interact with each other; the product and the world (Bonabeau Reference Bonabeau2002), these interactions are shown by the dashed arrows labelled 1, 2 and 3 in Figure 1. Agents may influence each other, such as when an agent has a poor experience with a product and their attitude towards the product becomes more negative, and dissuades, indirectly or directly, other agents from adopting the same product. The world may influence agents through stimuli defined in the scenario model, such as by changing transportation conditions available to agents to move around the world. Finally, agents may affect the world such as when an agent emits pollutants into the modelled environment. Data about the population’s beliefs and behaviours is integrated into variables held by individual agents. Distributions of the behaviours and beliefs of the agent population should be validated against the original survey data collected early in the process. The purpose of the ABM and framework is to ascertain the impact of the product, and in order to do that the rules regarding an agent’s choice to adopt the product are paramount.

Choice models have been used previously in product development (Li & Azarm Reference Li and Azarm2000), social interactions (Brock & Durlauf Reference Brock and Durlauf2001) and ABM (He et al. Reference He, Wang, Chen and Conzelmann2014). To isolate the effect of different product parameters on the social impact a binomial choice model is used, where only one type of product is available at a time of simulation. A multinomial choice model, where multiple variations of a product are available at one time, may more closely mimic real-world markets, but it becomes more difficult to interpret the impact results from a single product. An agent’s choice of adoption can be modelled as a utility function as shown in Eq. (1), where $ {U}_i $ is the product utility as understood by the agent $ i $ and the utility is calculated for each of the $ n $ agents. Agent $ i $ will adopt the product if $ {U}_i>0 $ . $ {U}_i $ contains an observed portion represented by $ {V}_i $ and an unobserved portion $ {\varepsilon}_i $ . The observed portion includes the influence of all product and TPB parameters where $ j $ is the number of parameters as shown in Eq. (2). The unobserved portion includes anything that may have been not included in the model as well as any error. $ \beta $ weightings may vary for individual agents or may be constant for the entire population. Varying $ \beta $ across the population shows different weightings of the factors. For example, $ \beta $ may vary across agents in the population if their preferences are influenced by age. The $ S $ variables will vary based on the conditions of the model and by agent interactions. By assuming a logit model, the unobserved portion of Eq. (1) is assumed to follow a logistic distribution (Greene Reference Greene, Mills and Patterson2009). This allows for simplifying the probability of the agent $ i $ adopting the product can be defined by Eq. (3). Therefore as the model progresses and conditions change, the probability of an agent adopting the product can change due to changes in values of the $ S $ terms of Eq. (2):

(1) $$ \hskip-.2pc {U}_i={V}_i+{\varepsilon}_i\hskip30em i\hskip2pt \in \hskip2pt \left\{1,2,\dots, n\right\}, $$

(2) $$ \hskip-1pc {V}_i={\beta}_{1i}{S}_{1i}+{\beta}_{2i}{S}_{2i}+\cdots +{\beta}_{ji}{S}_{ji}\hskip2em \forall j\hskip2pt \in \hskip2pt \left\{1,2,\dots, m\right\},\hskip4em i\hskip2pt \in \hskip2pt \left\{1,2,\dots, n\right\}, $$

(3) $$ \hskip0pc {P}_i=\frac{1}{1+{e}^{-{V}_i}}\hskip29em i\hskip2pt \in \hskip2pt \left\{1,2,\dots, n\right\}. $$

2.6. Framework: evaluation of the impact

As agents adopt the product, they will experience an impact. For example, if a product improves the health of an agent, that agent may be more likely to continue working, going to school, or survive to the next period. Measurements of aggregate impact across the population should be recorded for each run of the simulation. Depending on the model it may also be important to break down the impact by different demographic categories rather than a single measure of impact for the entire population. Product parameters can be iterated within the framework to see how changes in product parameters influence the impact of the product on society. It is important to note that because of the stochastic nature of an ABM, results will vary so it is important to run the model sufficient number of times to understand the distribution of results. By continuing to iterate on the parameters, a design team can understand what factors are most important to improve to maximise potential positive impact and minimise potential negative impact.

3.1. Illustration: product model

Mask attributes of effectiveness, comfort and aesthetics were selected for modelling. An assumption was made that the price of a mask was not a significant factor for mask adoption in the United States given the generally low cost of a face mask. Five mask designs with different attributes were evaluated in the model: (i) a control case with no masks, (ii) an N95 mask with high effectiveness but low comfort, (iii) a cloth mask with medium effectiveness and comfort, (iv) a neck gaiter with low effectiveness but high comfort and (v) an ideal mask that is very effective, comfortable and aesthetic. Effectiveness was rated on a 0–5 scale, with 0 being no mask and 5 a mask that filters $ 100\% $ droplets. Comfort was rated on a −5 to 0 scale with 0 being no mask, with an assumption that all masks will be less comfortable than no mask. Aesthetics were rated on a −5 to 5 scale with 5 being a mask with excellent aesthetic qualities that will make a user want to wear it, while a score of −5 will deter use. Attributes for the effectiveness of face masks were built from data by Fischer et al. (Reference Fischer, Fischer, Grass, Henrion, Warren and Westman2020) and Clapp et al. (Reference Clapp, Sickbert-Bennett, Samet, Berntsen, Zeman, Anderson, Weber, Bennett, Program and Epicenters2021). The authors acknowledge that the real-world effectiveness of a mask may not match those of a laboratory study, but for this exercise, it was assumed that the real-world effectiveness of a mask at reducing the risk of COVID-19 matched the filtration effectiveness from the literature. Data on the percent of the population that uses face masks correctly varies widely from 25% (Cumbo & Scardina Reference Cumbo and Scardina2021) to 90% (Cohen et al. Reference Cohen, Talarowski, Awomolo, Han, Williamson and McKenzie2021). For the illustration, it was assumed that 80% of agents who adopt a mask will use it correctly. If an agent was not wearing the mask correctly the efficacy of the mask was assumed to be cut in half. The attribute scores of each mask case can be seen in Table 1.

Table 1. Mask attributes

3.2. Illustration: society model

The goal of the society model is to fully define the model population with demographics, preferences, movement patterns and behaviours. Synthetic population attributes were based on Cape Elizabeth, ME and Angleton, TX. Each location will use a population of 10,000 agents. Cape Elizabeth has an older median age and a lower population density than Angleton. Two different locations were selected to observe how the results might change with different population demographics. The synthetic population was created using aggregate data from the 2019 US Census American Consumer Survey. The synthetic populations were created using an Iterative Proportional Fitting approach (Gargiulo et al. Reference Gargiulo, Ternes, Huet and Deffuant2010), used previously for simulation of infectious disease transmission dynamics in American Samoa (Xu et al. Reference Xu, Glass, Lau, Geard, Graves and Clements2017).

Assumptions and simplifications were made about the society to enable modelling such as all adults to travel to their workplace, and all children attend a school in person. The relative frequency of travelling to work, school and stores was defined with data from the American Time Use Survey (United States Bureau of Labor Statistics, 2020).

Relative user preferences were obtained through existing studies on mask use demographics (Igielnik Reference Igielnik2020) and perceptions (Howard Reference Howard2020). Older members of the population were assumed to have a higher preference for mask efficacy due to higher mortality rates for that demographic. The user preferences were used to obtain the utility function for a person $ i $ adopting the use of a face mask, see Eq. (4). Eq. (4) is then substituted into Eq. (3) to obtain the probability of the agent $ i $ adopting a mask on a given day in the model. The $ \beta $ coefficients are the relative weighting of different factors by an individual and the $ S $ terms are the scoring of that variable. Values and ranges for the variables of the utility function can be found in Table 2. The attitude towards mask scores, $ {S}_{ATB} $ , were initialized using a normal distribution with a median of −1.1, −1 and −0.9, respectively based on republican, independent or democratic political affiliation and standard deviation of 1 to simulate the population initially being more unfamiliar with face mask use (Igielnik Reference Igielnik2020). Political affiliation was chosen as a way to initialize attitudes towards masks instead of other mask use predictors such as a belief in science (Stosic, Helwig & Ruben Reference Stosic, Helwig and Ruben2021), due to the availability of political affiliation data for the synthetic population. Social norm scores were set based on the percent of connections in the agent’s network that have adopted masks with 5 being all connections have adopted masks and −5 being no connections adopting masks. Virus severity was defined by the percent of the population currently infected with the virus. These variables define the utility of a face mask for each user:

(4) $$ {V}_i={\beta}_{ATB}{S}_{ATB i}+{\beta}_{SN}{S}_{SN i}+{\beta}_{PBC}{S}_{PBC}+{\beta}_{Ei}{S}_E+{\beta}_{Ci}{S}_C+{\beta}_{Ai}{S}_A+{\beta}_V{S}_V+{\beta}_{AV}{S}_{AV}. $$

Table 2. Variables and coefficients for mask DCA

Note: Asterisks: For weights with multiple values: first value for individuals age 19–39, second value for individuals age 40–65 and third value for individuals age 65+.

Individuals in the network built social connections based on a small-world network structure (Watts & Strogatz Reference Watts and Strogatz1998) with their five closest neighbours and up to five individuals that share the same workplace. It was assumed that there was a 10% chance of deleting a social connection and forming a new connection with a random individual in the population.

3.3. Illustration: scenario model

The scenario model defines the scope and initial conditions of the simulation. The simulation was run for 5376-time steps equivalent to 8 weeks in the simulation. Each time step is equal to 15 minutes of simulated world time. Testing how face mask product parameters impact health and safety during COVID-19 was the main goal of the scenario. Initially, 25 people out of 10,000 were randomly infected with the virus. The SIR approach chosen for the virus allows agents to take on three conditions: (i) susceptible to the virus, (ii) infected with the virus and (iii) after 10 days they will recover or die (Cooper et al. Reference Cooper, Mondal and Antonopoulos2020). Parameters of the virus such as mortality rates by age, infectiousness and duration of illness were set according to data from the Centers for Disease Control and Prevention (COVID-19 Pandemic Planning Scenarios 2020). In this model, the scenario takes place in a closed system, therefore there are no new individuals to the population that may add cases of COVID-19. Parameters for mask mandates, the perceived behavioural control term, $ {S}_{PBC} $ in Eq. (4), were included on a 0 to 5 scale with 5 meaning that masks are required by law. A term for the availability of masks was included, the term $ {S}_{AV} $ in Eq. (4) on a scale of −5 to 5 with −5 being masks are extremely difficult to obtain and 5 being a supply of masks is provided to each household automatically. To isolate the mask design impact from policy impacts $ {S}_{PBC} $ , and $ {S}_{AV} $ were held constant throughout the simulations. $ {S}_{PBC} $ was set at 1, which would be equivalent to suggesting the public wear a mask and $ {S}_{AV} $ was set at 0 which would mean that masks are available but a supply is not given automatically to each household. These parameters would be useful if examining the influence of policy decisions on the social impact, but if the interest is only in the product parameters, they should be held constant.

3.4. Illustration: impact model

The focus of the Framework Illustration section is to demonstrate the framework used for social impact, but similar approaches could be taken with economic or environmental impact or all impacts combined. Health and safety were selected as the impact category to assess from among the categories presented by Rainock et al. (Reference Rainock, Everett, Pack, Dahlin and Mattson2018). To measure health and safety, indicators of the total number of COVID-19 cases and the number of deaths in the simulation were selected as the indicators to assess impact. The goal is to minimise both indicators. By viewing the results of these indicators for the different mask scenarios a design team can understand which product attributes are most important and may lead to the most positive impact.

3.5. Illustration: ABM

Most parameters of the ABM have been defined in the other input models (see previous illustration sections). The ABM for this illustration was developed using the Netlogo software package (Wilensky Reference Wilensky1999). To define how the virus can spread within the model, interactions between individuals were defined as two agents occupying the same coordinate in space. If this occurred there was a probability based on the COVID-19 parameters that the noninfected individual would contract the virus from the infected individual. Inputs to the model for the probability of influencing an agent’s attitude positively or negatively about masks were included to simulate media campaigns, news reports, and emerging evidence of mask use. This influence probability was held constant in the simulation to better explore the design space of mask parameters. Once at a given location, the agents would move in a random walk for the duration they were in a location such as a workplace, school or store to simulate random interactions that can take place throughout the day. By integrating the four input models in an ABM it is possible to model the health and safety impacts of face masks in the context of COVID-19.

3.6. Illustration: evaluation of the impact

To understand the range of results, 100 full simulations were completed for each potential mask design. A summary of the results is presented in Table 3. The results show that there is a statistically significant difference between all mask types and no masks with p-values <0.0001 from an analysis of variance test. A mask with low effectiveness, like a neck gaiter, still provides a significant reduction in COVID-19 cases but does not provide the same impact as a more effective mask. There was not a statistically significant difference in the number of COVID-19 cases between cloth, N95 and ideal masks with p-values ranging from 0.5 to 0.99 from a Tukey–Kramer test (Ramsey & Schafer Reference Ramsey and Schafer2012). There are differences in the results for each population. Angleton has a higher population density so it generally has more cases, while Cape Elizabeth with its older population has more deaths.

Table 3. Mask simulation results

Note: Each population contains 10,000 agents. Asterisks: The adoption % is the maximum adoption rate of face masks during the simulation.

To understand the individual mask attributes of effectiveness, comfort and aesthetics, a sensitivity analysis was performed using the cloth mask type as the baseline. This mask type was selected because cloth masks have become ubiquitous during the pandemic. Each of the mask attributes was improved by one unit while the others were held constant. Results from 100 runs of the simulation sensitivity analysis for each case are shown in Table 4.

Table 4. Sensitivity analysis

Abbreviations: A, aesthetics; C, comfort; E, effectiveness.

This sensitivity analysis shows that effectiveness is the most important attribute to improve by one unit to improve the positive impact. Effectiveness and aesthetics provided a statistically significant difference in cases, with effectiveness providing the greatest effect on the number of cases. With the limited resources that a design team has, the effectiveness attribute should be where the priority lies for the improvement of the design, as opposed to comfort or aesthetics.

The estimation of coefficients in the utility equation gives rise to a level of uncertainty. To understand the implications of uncertainty in the coefficients, propagation of uncertainty techniques was used (McClarren Reference McClarren2018). Each coefficient in the utility function was increased one at a time by 10% and the simulation was run 100 times. Results from these simulations can be seen in Table 5. Overall, these results show that the wrong estimation of a parameter by 10% does not change the output results of the simulation in a statistically significant way when comparing it to the base case of cloth masks using a Tukey–Kramer test (Ramsey & Schafer Reference Ramsey and Schafer2012).

Table 5. Propagation of uncertainty

This is how the framework would function in the design process. A model is built factoring in the product, society, particular scenario and potential impacts. As the design team iterates on product inputs in the model they can see how the estimated impact of the product changes. By observing these changes over iterations of product parameters the design team can see opportunities to increase the positive impact and reduce negative impact.

3.7. Illustration: verification and validation

It should be noted that this paper is focused on the theory of the framework shown in Figure 1 and the purpose of this illustration is to show how the framework could be implemented. All engineering models, even fundamental principles such as Hooke’s Law, are built on assumptions and approximations. Despite inexact results, models remain useful for estimating results, understanding the dynamics of a system and the relationships of parameters in that system to the results of a model (Epstein Reference Epstein2008). For those using ABM, it is important to be realistic and transparent about what is known and what is not about the model to conclude from the results. Due to gathering data on face masks from multiple sources without access to individual survey responses, assumptions were made for the scoring of coefficients in the utility function. Future work could include gathering necessary data about face masks in a single survey. The drawback of this is that opinions change over time. With that in mind efforts were made to validate the model. Although it is difficult to perform macro-validation of all model outputs, micro-validation and limited macro-validation were carried out as recommended by North & Macal (Reference North and Macal2007) and Wilensky & Rand (Reference Wilensky and Rand2015). There are seven recommended areas of validation for an ABM that were carried out, requirements validation, data validation, face validation, process validation, model output validation, agent validation and theory validation.

3.8. Illustration: model reusability

The time and resources required to build an ABM such as the COVID-19 face mask illustration can be prohibitive. By extending the use of a model to multiple products the cost of producing such a model may be more worthwhile. To test the reusability of models created from the framework, the illustration was extended to include the introduction of vaccines. Face masks and vaccines are closely related products where the models can be tested on the same target population with the same impact model. This enables the scenario, society and impact models to largely stay the same. Using models for more than one product should be a consideration during the model creation as deploying the model for subsequent products is generally more rapid.

The additional product parameter to assess for vaccine introduction is the number of doses that will become available each day. This would help inform decisions about scaling up manufacturing, and its impact on virus cases. To make these changes for the addition of the vaccine, a new utility equation for the adoption of the vaccine is created. It follows a similar form to Eq. (4) with some modifications as seen in Eq. (5). New variables are generated for the attitude towards the vaccine, effectiveness of the vaccine and extent of side effects. The variable descriptions and scores can be found in Table 6. Eq. (5) is substituted into Eq. (3) to determine the probability that an individual chooses to be vaccinated if doses are available. The values for the coefficients and scores were based upon a hypothetical vaccine that is 90% effective at preventing the contraction of the virus and has minimal side effects. Initial attitudes towards the vaccine were based on survey data that was assumed to have a normal distribution (Funk & Tyson Reference Funk and Tyson2020). The vaccine is administered to the oldest ages first and then to younger groups. Adult agents that made the choice to be vaccinated were selected at random if doses of the vaccine were available. The vaccine did not start being available in the model until 1 week had passed model time. Full efficacy for the vaccine required two doses in the model, and it was assumed that 80% of agents that received the first dose would go on to get the second dose. After one dose the vaccine was assumed to achieve half of its full efficacy. This scenario was run for 16 weeks of simulated time to better understand differences in the results due to the vaccine administration happening over time:

(5) $$ {V}_{nV}={\beta}_{ATBV}{S}_{ATBV i}+{\beta}_{SNV}{S}_{SNV i}+{\beta}_{PBCV}{S}_{PBCV}+{\beta}_{EV i}{S}_{EV}+{\beta}_{Ri}{S}_R+{\beta}_V{S}_V. $$

Table 6. Variables and coefficients for vaccine DCA

Note: Asterisks: For weights with multiple values: first value for individuals age 19–39, second value for individuals age 40–65 and third value for individuals age 65+.

Results from executing the framework with the addition of the vaccine can be seen in Table 7. Scenarios were tested with and without mask use. From the results, we see that the vaccine does help to reduce cases, but not as quickly as face masks that may be immediately available to the entire population. The best case is to have mask and vaccine use in the model.

Table 7. Mean simulation results for 100 model runs with vaccine introduction

Once again the purpose of the illustration of model reuse was to demonstrate how a design team might extend the model to new uses with the same society model. An initial model may take weeks or months to build depending on the data requirements, but if the model can be extended to a new use it may only take a matter of a few days or less to create the new model. Some model extensions are more difficult to implement than others. Social distancing measures, for example, would be more difficult to include because of greater complexities in the movement patterns of the agents in the simulated world.

4.1. Model results

The validity of results from predictive models has long been called into question (Epstein Reference Epstein2008) due to the difficulty of validating models. In spite of this difficulty, predictive models remain a useful tool and they continue to be used in many applications such as the COVID-19 pandemic (Wynants et al. Reference Wynants, Van Calster, Collins, Riley, Heinze, Schuit, Bonten, Dahly, Damen, Debray, de Jong, De Vos, Dhiman, Haller, Harhay, Henckaerts, Heus, Kreuzberger, Lohmann, Luijken, Ma, Martin, Andaur Navarro, Reitsma, Sergeant, Shi, Skoetz, Smits, Snell, Sperrin, Spijker, Steyerberg, Takada, Tzoulaki, van Kuijk, van Royen, Verbakel, Wallisch, Wilkinson, Wolff, Hooft, Moons and van Smeden2020) and climate change (O’Neill et al. Reference O’Neill, Oppenheimer, Warren, Hallegatte, Kopp, Pörtner, Scholes, Birkmann, Foden, Licker, Mach, Marbaix, Mastrandrea, Price, Takahashi, van Ypersele and Yohe2017). The validation of predictive ABM on innovation diffusion can happen for general models where the results can be validated against previous historical data. A key issue with this approach is that the creation of a general model that can be validated on historic data will likely not be specific enough for new innovations (Kiesling et al. Reference Kiesling, Günther, Stummer and Wakolbinger2012). Future work can be undertaken to validate the use of the framework on a new product or intervention and measure the product impact across a sample population.

Current strategies for handling the validation of ABM involve performing micro-validation techniques and limited macro-validation (Midgley, Marks & Kunchamwar Reference Midgley, Marks and Kunchamwar2007; North & Macal Reference North and Macal2007). Micro-validation can be used on individual components of the model that can be validated by the modellers or using previously validated components. Macro-validation can be carried out on a limited basis and may be restricted to qualitative validation (Bonabeau Reference Bonabeau2002; Moss & Edmonds Reference Moss and Edmonds2005). There are cases where experimental macro-validation is not possible, such as the case of modelling the integrity of a nuclear waste repository for 10,000 years, and good practice is to state assumptions clearly and justify them (McClarren Reference McClarren2018). Although the quantitative validation of ABM may be limited, there is still value in creating representations of a system to highlight places where opportunities may exist to maximise positive product impact (Wilensky & Rand Reference Wilensky and Rand2015).

Because of the stochastic nature of an ABM, results can vary for different runs of the simulation. It is important to have a sufficient number of runs to understand the distribution of results (Epstein Reference Epstein1999). In the case of the framework presented in this paper, a sufficient number of runs for each instance of product parameters needed to be executed. This enabled the use of statistical tests to understand where there is a statistically significant difference in the product impact between the different product parameters.

4.2. Resources required

ABM can be resource-intensive to implement and execute. A balance needs to be created between making a model that is simple enough to implement with the available resources and complex enough to provide meaningful results (Rahmandad & Sterman Reference Rahmandad and Sterman2008). Creating decision models requires detailed information about the preferences of individuals. Demographic and behaviour information is needed about the target population. In some countries, this information is accessible through government surveys but is not always available for a target area. In addition to the data about individuals, enough must be known about potential product designs to understand the potential product impacts. All of this combines to make the initial data collection for this framework time intensive. To lessen the resources required, it is recommended to use available data sets whenever it is appropriate.

Depending on the size of the models, they may also be computationally intensive to execute. Although this is not always possible to completely overcome, computational resources are more accessible than ever. Reducing the model to the minimum necessary scope helps balance the trade-off between accuracy and cost.

The resources required for implementation create a situation where this framework may not be feasible for small projects or products with limited distribution. For large-scale projects where products may affect many people, taking the time to use this framework to model the product impact will help inform design decisions to have better product impact outcomes.