2.1. Health example: Cancer case study

A typical policy issue addressed through an environmental health regulation is human exposure to a contaminant with negative health effects. The regulatory goal is to reduce the presence of the contaminant or reduce exposure to it. We walk through an example analysis of benefits for reducing exposure to a carcinogen—an example loosely based on the RIA for EPA’s methylene chloride rule (Environmental Protection Agency (EPA), 2019)—because detailed methods for risk assessment and benefits analysis for carcinogens have been established. The risk assessment methods have been subject to criticism (Nichols & Zeckhauser, Reference Nichols and Zeckhauser1988; Abt et al., Reference Abt, Rodricks, Levy, Zeise and Burke2010) due to significant uncertainties that arise, for example, from needing to extrapolate information from animal studies to evaluate risk to humans. Nonetheless, the analytic difficulties are even greater for many noncancer health effects, where scientific information in a form amenable to benefits analysis is often unavailable.

In the simplest form, benefits for the regulation of a carcinogen are calculated as:

$$ Benefits=\Delta\;Exposure\hskip0.3em \ast \hskip0.3em Unit\ Risk\hskip0.3em \ast \hskip0.3em WTP $$

The key information needed to calculate benefits is a baseline measurement of population exposure to the contaminant and an estimate of exposure after regulation. For a carcinogen found in the workplace, an analyst might estimate the baseline population exposure using data on workers engaged in activities where the contaminant is present and sample measures of the contaminant in the relevant environments. A regulation banning the contaminant might effectively drive workplace exposure to zero. Alternatively, a regulation might require the use of personal protection equipment. Estimating the amount that the equipment reduces exposure is possible because the equipment can often be tested to determine the amount of protection offered. In some cases, data collected during a case study involving a specific population may be sufficient to support statistical analysis of the exposure change, which can then be applied to a more general setting. For example, advances in environmental modeling and data collection have allowed for more intricate measurement of exposure changes due to a regulation (Currie et al., Reference Currie, Voorheis and Walker2021).

For regulation of a carcinogen, benefits analysis relies on a risk assessment to quantify the change in cancer risk expected due to changes in the levels of exposure to the contaminant. This change is “unit risk” in the equation above and is derived from a dose–response function. When the dose–response function is linear, its slope gives the excess risk of cancer to a population due to people’s exposure over their lifetimes. The risk is “excess” in that it is an increment beyond their expected risk without the contaminant in the environment.

Unit risk, or the slope factor, is the excess lifetime cancer risk estimated to result from continuous exposure to, or ingestion of, one unit of concentration of the contaminant. Unit risk multiplied by exposure yields the projected incidence of cancer. For example, an individual’s exposure for an airborne contaminant is calculated as:

$$ C\times EF\times ED $$

where C is the concentration of contaminant (e.g., μg/m³), EF is the exposure frequency (days per year), and ED is the exposure duration (years).

One can estimate the expected number of excess cancer cases from a contaminant by summing individual exposures to get the population exposure and multiplying that exposure by the unit risk. The reduction in cancer cases due to a regulation is obtained by conducting this calculation for the baseline exposure level without the regulation and for the exposure level expected once the regulation is in place.

The last step in estimating benefits is converting benefits in terms of the reduction in expected cancer risk to dollar values. For cases that would result in premature fatalities, the VSL is the appropriate weight. For nonfatal health effects, monetizing the effects using the avoided cost of illness is a common alternative to using WTP measures (Environmental Protection Agency (EPA), 2010a).

Health benefits due to regulating a carcinogen do not appear immediately because cancer generally has a latency period. Analysts account for this delay by discounting the estimates over the interval of time between reduced exposure to the contaminant and when the cancer would be expected to occur at the end of the latency period.

2.2. Safety example: FMVSS case study

The National Highway Traffic Safety Administration (NHTSA) regularly issues FMVSS rules to implement laws from Congress and has a well-established framework for conducting benefits analysis for these rules. For this example, based loosely on the RIA for National Highway Traffic Safety Administration’s (NHTSA’s) (2015) Electronic Stability Control Systems on Heavy Vehicles rule, we consider a type of FMVSS that targets crash avoidance and aims to reduce the probability of a crash altogether. Preventing a crash reduces the associated deaths and injuries, which is the primary source of benefits for this type of rule. Other types of FMVSS target crashworthiness and aim to reduce the probability of injury or death for vehicle occupants if a crash occurs. The analyses for both types of rules are similar in terms of how they translate into a reduction in the probability of deaths and injuries.

In benefits analysis, a key difference between FMVSS and carcinogen regulations is the use of actual deaths and injuries versus inferred deaths and injuries. The typical FMVSS regulation targeting crash reduction starts with a “body count” of the deaths or injuries due to the hazard. This regulatory problem differs from the case of an environmental regulation in that the impacts are directly observable in epidemiological data. The “body count” associated with carcinogen regulations, in contrast, is inferred or calculated indirectly (as discussed in Hammitt et al., Reference Hammitt, Morfeld, Tuomisto and Erren2019) from baseline exposure levels in the population and the dose–response relationship.

For an FMVSS rule, NHTSA calculates benefits using a formula with the general form:

$$ Benefits= Target\ Population\hskip0.3em \ast \hskip0.3em Effectiveness\hskip0.3em \ast \hskip0.3em WTP $$

The target population is measured as the actual fatalities and injuries to individuals involved in crashes in the preregulatory environment or baseline. Crash fatality data come from the Fatality Analysis Reporting System, a census of crashes involving fatalities on highways.Footnote ⁴ Injury estimates come from the National Automotive Sampling System/General Estimate System, a nationally representative sample of police-reported motor vehicle crashes in which trained data entry personnel interpret and code data.Footnote ⁵ Some of these fatalities and injuries would be eliminated, in theory, if the vehicles had the technology mandated by the regulation and had not crashed. Determining which crashes could have been prevented involves an engineering judgment. For a crashworthiness technology, identifying a target population may be more complicated because the crash is not assumed to be prevented; instead, the severity is presumed to be reduced. As with crash avoidance, determining reductions in fatalities and injuries involves an engineering or other professional judgment.

Some FMVSS analyses have a complication in that the required technology may have already been adopted by some percentage of the regulated vehicles. In this case, the target population will need to be adjusted to estimate the fatalities and injuries that would have occurred if no vehicle had installed the required technology.Footnote ⁶

The effectiveness rate for the technology is the key parameter in a safety benefits analysis and typically the most difficult parameter to estimate. For an FMVSS safety regulation, the sources of data and information for constructing an effectiveness measure are robust. Effectiveness rates might be based on computer simulations, expert panel assessments of crash data, and research experiments. For rules addressing crashworthiness, establishing an effectiveness measure often involves crashing vehicles, examining impacts to crash test dummies, and extrapolating those results to real-world crash scenarios. In the case of partial adoption of a technology, effectiveness could be derived from a statistical analysis of real-world crashes involving vehicles with and without the technology.

The effectiveness rate represents a synthesis or meta-analysis of many sources of data and is the key to quantifying the physical relationship between regulatory inputs. It serves the same purpose as the dose–response relationship in benefits analysis for the regulation of environmental contaminants. While methods for deriving an effectiveness rate measure for an FMVSS rule are well-established, this is not the case for other types of safety regulations, particularly in cases with limited epidemiological data on human injuries and deaths.

The final step in estimating benefits is converting the reduction in deaths and injuries to dollar values. For avoided crashes that would result in a premature fatality, the value equals the VSL. For avoided injuries, analysts approximate WTP values by applying a fraction of the VSL depending on injury severity (U.S. Department of Transportation (DOT), 2021).

Benefits due to reduced crashes accrue over the lifetime of vehicles once they are produced with the required technology. The benefits phase in, however, only as vehicles without the required technology are retired and replaced. Analysts account for the accrual and phase-in by discounting the monetary weight over the appropriate time intervals.

2.3. Comparing health and safety benefits analysis

The example regulations show that benefits analyses for the example health (carcinogen) and safety (FMVSS crash avoidance) regulations share many similarities. The goals for both types of regulation involve reducing the risk of fatal and nonfatal effects. For regulation of a carcinogen, the mechanism for achieving the goal is the change required by the rule to reduce exposure to a contaminant, which reduces cancer risk. The reduction in cancer risk provides an estimate of the number of cancer cases projected to be avoided due to the rule. This estimate can then be monetized according to existing values. For cases that would result in premature mortality, an analyst would use VSL; for cases that would result in morbidity (illness but not death), an analyst would use a cost of illness or WTP alternative if available. For an FMVSS crash avoidance rule, a requirement to install safety equipment is the mechanism to reduce the risk of a motor vehicle crash, which serves as the basis for estimating the number of fatalities and nonfatal injuries that would have been prevented if existing vehicles had been equipped with the equipment. Avoided fatalities are valued using VSL and nonfatal injuries are assigned fractions of the VSL, depending on injury severity.

The comparison shows one key difference in the basic data the two types of rules use for quantifying impacts, particularly for characterizing the baseline unregulated state. For a health regulation, the risk to be reduced is not measured directly in terms of number of cancer cases that are occurring and to be addressed within the context of the specific regulation. Rather, the level of exposure to the contaminant is measured, with the number of excess cancer cases (and associated morbidity or mortality) calculated using the dose–response relationship. A safety benefits analysis, in contrast, is an actuarial or forensic approach because it is grounded in past fatalities. The measurement of risk for a safety regulation like FMVSS begins with the historical record: the actual count of fatalities and injuries that are occurring and to be addressed by the regulation. This approach to estimating safety benefits can give the impression that impacts are more tangible for safety versus health regulations. However, it also creates a limitation for evaluating emerging safety risks that have yet to appear in the historical record.

While the actuarial approach offers a useful check for evaluating the plausibility of estimated outcomes, considerable subjective judgment is exercised in identifying injuries and deaths that would have been prevented with the technological intervention under consideration. In addition, the practice of grounding risk analysis in past fatalities creates another set of difficulties because not all safety problems in regulation can be reduced to fatalities. When a safety problem does not have a historical record, measuring economically meaningful outcomes is difficult, and the same problems described in health benefits analysis by Dockins et al. (Reference Dockins, Griffiths, Owens, Simon and Axelrad2004) occur in safety benefits analysis.

Table 1 summarizes approaches to benefits analysis in the example health and safety regulations. If the underlying risk assessment yields a dose–response relationship or an effectiveness measure, then the benefits analysis for both is similar. If data are insufficient to support dose–response modeling or effectiveness measurement, then benefits analysis would have similar problems of unmeasurable and unquantifiable impacts.

Table 1. Comparison of regulatory analysis for rules involving carcinogens and FMVSS.

4.1. Effectiveness measurement for fatigue risk management and training

Accident studies usually identify multiple causes contributing to a single accident and often uncover behaviors or contributing factors that are common to a wide range of accidents. Lack of training and transportation sector employee fatigue are two of the most cited contributing factors across a wide range of accident types and modes of transportation.

Fatigue is a condition widely recognized to increase accident risk, and higher levels of fatigue increase that risk. Reduction of fatigue is a common objective of safety regulation. The linkage between fatigue and increased accident risk has been studied sufficiently to provide an empirical basis for estimating accident reductions expected from measures that address employee fatigue in the transportation sector.

The Federal Railroad Administration (FRA) examined historical fatigue-related accidents for the Hours of Service of Railroad Employees rule (Federal Railroad Administration (FRA), 2011). Accidents involving fatigue are human factor accidents that involve an error on the part of a train crew member, like not stopping at a red signal. FRA requires railroads to report rail accident and incidents involving damages above a certain monetary threshold ($11,200 as of January 2021). The reports include location, type of accident (e.g., derailment, collision), cargo type, damages incurred, and accident causes. FRA identified 616 human factor accidents involving 723 injuries and 8 fatalities during the 10-year period from 2000 to 2009 as potentially fatigue-related.

To estimate the number of human factor accidents that occurred under conditions of fatigue, FRA applied the results of a study of train crew work schedules and fatigue (Federal Railroad Administration (FRA), 2008). The study collected 30-day work histories of 2800 locomotive crew members involved in 1400 accidents and applied a biomathematical fatigue model to assess how work schedules contribute to increased fatigue and elevated accident risk. The modeling approach assigned fatigue scores through each worker’s shift including the time the worker was reported to have been involved in an accident. From all work histories combined, the approximate proportion of work time spent by workers at each fatigue level can be computed as well as the proportion of accidents occurring at any fatigue level. This information allows for calculating relative risk of an accident as a function of fatigue level and is like a dose–response function applied in modeling cancer risk.

Using the study’s analysis, FRA estimated that the measures to address fatigue considered in the rule would decrease the risk of a human factors accident by 67%. This reduction, when applied to the historical number of accidents (as well as the deaths and injuries that occurred due to those accidents), yields an estimate of the benefits of the rule. FRA made a further adjustment to account for its Positive Train Control (PTC) rule, which was expected to prevent 80% of PTC-preventable accidents (so that 20% of accidents remain), to get a net decrease of 13.3% (0.67 * 0.20 = 0.133).

Like employee fatigue, lack of training is often cited as a contributing factor to accidents. While training on certain tasks would logically seem to encourage adoption of safer practices that reduce the risk of an accident, demonstrating and quantifying the relationship between safety and training are difficult.

In the analysis for the Pilot Professional Development rule, the Federal Aviation Administration (FAA) cited several instances of accidents where experts considered unprofessional pilot behavior to be a contributing factor (Federal Aviation Administration (FAA), 2020). The purpose of the rule was to require behavioral training that the FAA determined was likely to reduce the risk of pilots engaging in problematic behaviors that increase the risk of an accident. However, it was not possible to separate the pilot mistakes due to unprofessional behavior from the technical mistakes that were also contributing factors in the examined accidents. Because it was not possible to measure the relationship between training and the probability of an accident, FAA limited the benefits analysis to a qualitative discussion.

FMCSA encountered similar difficulties quantifying the relationship between accident risk and training in the Entry Level Driver Training rule analysis (Federal Motor Carrier Safety Administration (FMCSA), 2016), despite widespread agreement among stakeholders that the training required by the rule would likely to lead to improvements in safety outcomes. The belief in training as a mechanism for improving safety was reflected by companies that would provide such training to their employees even without a requirement to so. While the companies may have had anecdotal evidence that employee training would increase safety, the information was insufficient to support quantitative analysis. The analytical problems encountered included lack of control groups; inability to control for driver-specific factors such as age, experience, and exposure; small sample sizes; and poor generalizability of results.

FMCSA could quantify the effect of training on fuel consumption and repair and maintenance costs. Expenses for fuel and repair and maintenance directly affect a company’s bottom line, probably more so than accidents, which remain relatively rare and are typically covered by insurance. Companies have greater incentive to track the effects of training on factors that directly affect profitability, especially if they believe overall safety performance is adequate. Quantifying these nonsafety benefits allowed FMCSA to calculate the reduction in crash rate needed for the rule to break even or for benefits to equal costs. This break-even analysis indicated that, if the rule resulted in a 3.61% reduction in crashes (central case), benefits would equal costs.

In sum, while it is reasonable to expect that measures to reduce fatigue and improve employee training would lead to improved safety outcomes, demonstrating a quantitative relationship is not always straightforward. For fatigue, it is necessary to measure employee fatigue levels throughout the work shift and the probability of an accident at each fatigue level. It is also necessary to measure quantitatively the reduction in fatigue expected due to the regulatory intervention. All of this is possible, as FRA has demonstrated, but methods for quantifying the effect of training on safety have yet to be developed.

4.2. Benefits of Safety Management Systems

Studies of major accidents show that they are the consequences of multiple, smaller failures. The “Swiss cheese” model of human error is a framework that describes the multiple, smaller failures that lead up to a major accident or catastrophic failure (Reason, Reference Reason2000). The likelihood of the same circumstances repeating and causing the same accident is essentially zero, but the likelihood of preventing a similar set of circumstances leading to a different accident is probably much greater than zero. Taken to the extreme, failure to identify and treat these smaller failures can result in the same hazards becoming recurring “near-miss” events that cause people and organizations to under-estimate risks and reinforce risky decisions (Tinsley et al., Reference Tinsley, Dillon and Cronin2012).

One way to reverse risky behavioral tendencies created by “near-miss” accidents is by establishing a Safety Management System (SMS). An SMS is a structured and documented set of procedures allowing company personnel to implement safety policies (Li & Guldenmund, Reference Li and Guldenmund2018). The objective of an SMS is to proactively manage safety by identifying potential hazards and their risk and then by implementing risk mitigation measures before any accident occurs. An SMS is updated continuously and evolves based on observations of current work practices and recognizing the need for changes or additional safety protections. An SMS is thought to reduce safety hazards and incidents by creating a safety culture or an awareness of and commitment to safety shared by employees. The presence of an SMS alone could create benefits like increased employee satisfaction and operational efficiencies, but these are difficult to measure. In addition, a culture of safety culture can reduce human error and, in turn, reduce the probability of an accident and resulting damages. The latter category of benefits, which have concrete metrics like reduction in fatalities and injuries, typically is the focus of SMS benefits analysis.

In 2015, FAA required scheduled air carriers to establish and maintain an SMS in the Safety Management Systems for Domestic, Flag, and Supplemental Operations Certificate Holders rule (Federal Aviation Administration (FAA), 2015). The FAA based its benefits analysis on 123 human error accidents that the National Transportation Safety Board (NTSB), an independent federal agency with a congressional mandate to investigate major accidents, had investigated. In the investigations, NTSB identified probable causes for the accidents, such as flight crew error; the failure of industry to incorporate hazard data and follow procedures; and inadequate employee oversight. FAA’s Office of Accident Investigation and Prevention reviewed the NTSB reports and assigned a probability that each would have been averted if an SMS program had been in place. The assigned probability came from expert judgment rather than a statistical or other type of formal evaluation and for most accidents was 20%. The expert judgment of the reduction in probability of an accident serves the same purpose as an effectiveness rate and is applied to the total number of fatalities, injuries, and damaged aircraft associated with an accident to estimate benefits.

A few years later, the Federal Transit Administration (FTA) required transit agencies to implement an SMS in its Public Transportation Agency Safety Plans rule (Federal Transit Administration (FTA), 2018). The RIA cited several major accidents which resulted in multiple fatalities and injuries, as well as significant property damage. The NTSB identified multiple factors as probable accident causes, including operator fatigue; communication failures; lack of safety briefings for employees; inadequate safety procedures for carrying out work; and inadequate safety oversight by the transit agencies. Other probable accident causes included employee errors such as train operators not following rules and track inspectors not looking out for oncoming trains while repairing the tracks (U.S. Government Accountability Office (GAO), 2011). NTSB concluded that, had an SMS been in place, several of these accidents could have been avoided or reduced in severity. Thus, NTSB recommended that transit agencies adopt an SMS as required in the FTA rule. Unlike FAA, FTA could not assign a probability that an accident would not have occurred with an SMS in place; instead, FTA discussed benefits qualitatively.

The benefits analyses for SMS rules raise analytical issues. Disentangling the generalized impacts of SMS from the impacts of additional risk mitigation measures like employee training or improvements in operational procedures is difficult. Also, little attention has been given to the organizational benefits of SMS. Benefits like increased employee satisfaction or improved operational efficiencies might be useful for justifying SMS programs on their own merits, without needing to predict or speculate about the additional actions that translate into a reduction in accident probability. In addition to studies measuring the effectiveness of SMS on outcomes like employee satisfaction and efficiency, economic studies to value the effectiveness would be necessary.

The effects of SMS on accident probabilities depend upon the mitigations or actions that a business would need to undertake to prevent the accidents from happening. A reduction in probability of an accident does not occur due to the presence of SMS alone; the business must also identify risks and undertake further actions to mitigate the risks through the SMS process. In addition, without knowing the specific risk mitigation measures, it is not possible to estimate the full cost of an SMS before implementing it. The uncertainty in measuring SMS costs limits the applicability of other tools that could provide insights into benefits when the lack of an effectiveness measure prevents quantitative analysis. For example, break-even analysis can identify the number of accidents that would need to be prevented to balance benefits against costs, but it does not provide much insight for SMS programs because of uncertainty in measuring costs. When costs are uncertain or cannot be fully estimated, it is not possible to calculate the break-even value.

4.3. Low-probability, high-consequence events: multiple causes and interventions

Some types of safety-related accidents occur infrequently but have catastrophic consequences in terms of death and injury, generating considerable regulatory interest. While estimating the probability that the accidents will occur is difficult when they do not occur in most years, benefits analyses for regulations that aim to prevent these low-probability, high-consequence events often try to follow the historical record framework used for more common events. Accident investigations often identify specific causal and contributing factors for the events, and experts use that information to judge the degree to which alternative procedures could have prevented the accident. If possible, this judgment is expressed as a direct estimate of the reduction in the probability that the accident would have occurred if the alternative procedures had been followed. Adoption of the probability-reducing procedures may then be recommended as new regulations. Depending on the severity of the instigating events, the recommendations can become Congressional mandates for regulation.

Because safety benefits analysis tends to be anchored to a historical record, a risk assessment often begins with identifying causes of specific accidents and considering whether a specific remedy could have prevented the accident. The baseline scenario, or “world without regulation,” is the world as it already happened, and the hypothetical policy scenario asks how different that world would have looked if the regulation had been in place. Forming the hypothetical policy scenario is complicated because a specific remedy will rarely be completely effective at addressing an accident cause. For example, inadequate rest may be identified as one of several causes contributing to an accident, but a rest requirement might not have necessarily prevented the accident from occurring. No measure is guaranteed to eliminate the problem of human fatigue; even if it could, other contributing factors may remain untreated, and the accident might still have happened.

In its Flight Duty and Rest rule, FAA relied on experts to estimate the probability that each of a set of historical crashes would have been prevented if the rule’s requirements had been in place at the time of the crash (Federal Aviation Administration (FAA), 2011). FAA calculated expected benefits by multiplying these expert-based probability estimates by the economic costs of the actual safety consequences and adding the results across all relevant historical events (Table 3).

Table 3. Benefits analysis in the FAA flight and duty rule.

Airline crashes are rare and typically occur due to multiple contributing factors. A single crash or set of crashes might result in several separate regulations independently aimed at addressing the identified causes of the crash. For example, FAA’s rule affecting pilot training requirements (Federal Aviation Administration (FAA), 2020) drew upon a similar pool of accidents as its duty and rest rule. FAA accounts for the use of the same crashes for multiple rules independently addressing the causal factors by ensuring that the sum of the crash reduction probabilities does not exceed one. For example, Colgan Air Flight 3407 crashed in Buffalo, NY in 2009, and the NTSB accident investigation identified several probable causes of the crash. FAA used the crash as the basis for benefits estimation for four rules, assigning crash reduction probabilities of 0.5, 0.2, 0.2, and 0.05.Footnote ⁷ With all four rules in place, the expected reduction in probability that the crash would have occurred is 0.95.

The FAA approach to estimating risk reductions and benefits could be applied in other settings where one or more large accidents give rise to a suite of regulations. Analysts will need foresight regarding the actions that will be undertaken to address individual contributing factors as well as estimates of their relative importance.

4.4. Emerging risks and lack of a historical record

When accidents are rare or due to multiple causes that cannot be readily disentangled, focusing on past events limits the degree to which risk assessments can yield effectiveness measures that analysts can use in a quantitative benefits analysis. When the regulatory problem involves a new and emerging technology for which no historical record exists, the problem is compounded, and safety benefits analysis will need to shift from its forensic nature to a more forward-looking or prospective approach. This is the case for “enabling regulations,” which expand production or consumption opportunities by revising existing regulations to allow new or previously prohibited or limited economic activity.

Enabling regulations may remove barriers to adopting Footnote emerging technologies but do not preclude establishing some restrictions to manage emerging and mostly unknown risks. FAA’s Part 107 rule, for example, expanded options to allow individuals to obtain a pilot certificate to operate unmanned aircraft (Federal Aviation Administration (FAA), 2021). The RIA focused on the new economic opportunities created by the rule: the rule lowered costs to pilots already operating small UA and considerably reduced costs to entry, which would attract new pilots to the industry and expand business opportunities. Similarly, FRA treated the creation of a new high-speed rail equipment tier (Tier III) as a business opportunity that would facilitate implementation of high-speed rail and increase the probability that a system would be completed in the USA in the next 30 years by 10–25% (Federal Railroad Administration (FRA), 2018). The RIAs for these rules did not formally consider the safety impacts of allowing these emerging technologies.

Safety benefits analyses for enabling rules are more speculative than analyses for known and experienced hazards. With no historical basis for calculating event probabilities and perhaps only a vague notion of potential consequences, quantitative safety benefits analysis may not even be possible. In this case, qualitative analysis may be the best approach, as in FAA’s analysis of its Operations over People rule, which extended the Part 107 rule by allowing routine operations of unmanned aircraft over people and routine operations at night.