11.3.1 Probabilistic, qualified evidence

In the introductory example above, the evidence for the effectiveness of a management intervention was measured as a probability. In terms of providing evidence, this is a routine way in which a scientist would express their recommendation. However, for implementing management, this can be problematic. For instance, telling a manager that there is a 70% chance that the intervention will work is only partly addressing the question of the manager, namely should they undertake the action or not? How is a manager to know whether their particular circumstances are likely to lead to them being in the 70% of cases in which the intervention works or in the 30% in which it fails?

In this context, the meaning of probabilities conveyed by scientists may not always be fully clear. Consider an everyday example. We might be told by a weather forecaster that there is a 50% chance of rain today. However, the meaning of that probability is not typically explained. Here are four interpretations.

1. (i) It will either rain everywhere or nowhere: it could be one or other of these outcomes, for example, because it is not possible to predict the precise location of a weather system.

2. (ii) It will rain for 50% of the time during the forecast period: for example, there are patchy rain clouds that are continually moving.

3. (iii) It will rain in 50% of places: for example, there are rain clouds cover 50% of the area that do not move.

4. (iv) The forecaster is unable to tell us whether it will rain or not and is telling you to flip a coin.

The technical interpretation of a probability in a weather forecast is that this probability represents the fraction of times a given outcome (e.g. raining within a defined set of areas) occurs in a set of stochastic realisations. This definition, interestingly, can incorporate all four of the above interpretations. Nevertheless, the probability quoted is a form of knowledge uncertainty that has a very specific meaning: it is a measure of model uncertainty/variance.

This highlights a second aspect of scientific evidence that is problematic from the perspective of management, namely that scientific evidence is usually qualified. The statement ‘there is a 50% chance of rain’ from a scientific perspective should also be qualified by the statement ‘across a set of simulations, given the assumption that the model is correct’. If the model is wrong then the prediction could be greatly different.

The task of a manager is to convert such evidence into action (i.e. the binary outcome of whether to act or not). As noted in the introduction, the decision then involves costs and benefits, defined in the widest sense and including values. To continue the hypothetical example, carrying an umbrella is low cost and high benefit, so a 50% chance of rain would render this a good strategy. On the other hand, a manager who is spraying a pesticide requires good conditions, and a 50% chance of rain would potentially carry an unacceptable risk that this costly action (in terms of fuel, time and chemicals) would fail.

11.3.2 General versus situational outcomes

The aim of science is typically to find answers that are as general and robust as possible. A scientist faced with evaluating the effectiveness of a management intervention will attempt to find whether there is evidence of its effectiveness, on average, and then probably focus on understanding the mechanisms that drive it. In contrast, a manager is faced with the task of managing a given site over a defined time period. There is a potential conflict between these perspectives, as the scientific perspective typically averages over variation arising from site-specific variations, whereas this is precisely the variation that a manager is focused on. For a scientist, the local variation at a specific site is essentially nuisance variance.

Although perhaps something of a caricature, there is undoubtedly a real problem in addressing these differences in perspectives. The situation is complicated by the difference in success measures for scientists and managers: scientists prove success by presenting results that are of interest to a wide range of others and that do not focus on specific instances (e.g. in scientific papers); managers measure success based on the state of their site. This difference in perspectives is reflected in the contrasting ways that scientists and managers treat uncertainty. From the science perspective the variation around the mean is a quantity that is to be minimised where possible; in contrast, a manager needs to know where their site sits with respect to this variation, and whether local circumstances render the overall average outcome pattern inapplicable.

11.4.1 Ignoring uncertainty should not be an option

One of the most important causes of uncertainty is lack of information. This is particularly an issue when information is lacking on rare and difficult-to-observe species, meaning that clade-wide conservation assessments are potentially compromised. The International Union for Conservation of Nature (IUCN) is an important organisation that collates data on the conservation status of species from a wide range of taxa into a set of threat states (Mace & Lande, Reference Mace and Lande1991). This extensive and important exercise informs conservation strategies in a range of contexts (Rodrigues et al., Reference Rodrigues, Pilgrim and Lamoreux2006). The basis for the assessment is a five-point scale of threat status for wild extant species. Species are classified as Least Concern (LC), Near Threatened (NT), Vulnerable (VU), Endangered (EN) or Critically Endangered (CE). Extinct in the Wild and Extinct are categories of extinction beyond these five points, representing species loss.

The amount of data required to apply these criteria varies between taxa. In some cases the amount of information required is quite low. For example, the Nechisar nightjar (Camprimulgus solala) is classified as VU despite being known from only a single wing and a single sighting. On the other hand, for some groups (e.g. mammals and amphibians) the data requirements for the assignment of conservation status are more exacting. Those species for which sufficient information is not available are assigned a status termed Data Deficient (DD). The number of DD mammal species is a considerable fraction of the group (483 of 4186 species; i.e. >10%) of mammals studied by Jetz and Freckleton (Reference Jetz and Freckleton2015).

Denoting species as DD is, effectively, a way of dealing with uncertainty. It is essentially the same as ignoring missing data in an analysis. This way of dealing with data uncertainty is, however, fraught with pitfalls, and a large literature exists on dealing with missing data and associated uncertainty (Nakagawa & Freckleton, Reference Nakagawa and Freckleton2008). It is well understood that non-randomness in the pattern of ‘missingness’ can yield highly misleading analyses.

In the case of conservation assessments, the concern with DD mammal species is that the factors that drive data deficiency are closely related to those that determine extinction threat. For instance, if species are difficult to observe it is likely to be because they only occur at low density in remote locations, or population trends are unknown because they are so rare. It is easy to see that this set of criteria could lead to species being ignored from conservation assessments even though they are threatened.

Jetz and Freckleton (Reference Jetz and Freckleton2015) tested this hypothesis by applying a framework for phylo-spatial modelling of IUCN threats, then using this to predict the probability that DD species are threatened. Species that are DD are predicted to have much higher threat probabilities than those that have been assessed already (Figure 11.1). The fraction of threatened mammal species is therefore underestimated by the current system of assessment.

Figure 11.1 The importance of dealing with uncertainty in conservation assessments. We used models to generate threat probabilities for mammals. (a) These probabilities do an effective job of distinguishing species that are Least Concern (green bars) from those that are Critically Endangered (orange bars); (b) our models were used to predict threat probabilities for species that were Data Deficient (DD) (pink bars) compared to species that were assessed (grey bars) (i.e. to reduce uncertainty in assessment).

Interestingly, the same is not true of birds (Lee & Jetz, Reference Lee and Jetz2011), as a much smaller fraction of them are considered DD because a lower threshold of information is required to assess threat status. Thus, the recent taxonomic explosion that has led to the creation of 1000 new species of birds (del Hoyo et al., Reference del Hoyo, Collar and Christie2014, Reference del Hoyo, Collar and Christie2016) has not resulted in 1000 species being assigned to the DD category.

This example illustrates an important point about uncertainty that is relevant to conservation and management. Ignoring uncertainty by simply excluding cases where data are missing runs the risk of introducing bias and so, in general, should be addressed if at all possible (Millner-Gulland & Shea, Reference Millner-Gulland and Shea2017). In the introduction I noted that the likelihood of implementing an action is low, irrespective of its actual effectiveness, when there is great uncertainty associated with its effectiveness (i.e. the parameter p is low). In this example, data-deficiency data result in no action being taken (p is low because of uncertainty), although the evidence (Figure 11.1) is that the intervention (assigning status of ‘threatened’) is justified with high probability.

11.4.2 Providing more data/evidence

The preceding example highlights that, where possible, additional data should be used to plug gaps in knowledge. One of the ways that scientists tend to qualify conclusions (see Section 11.3.1) is to say that we cannot be confident because more data are required. As argued by Millner-Gulland and Shea (Reference Millner-Gulland and Shea2017), this can prevent effective management-relevant advice being given.

The example from Jetz and Freckleton (Reference Jetz and Freckleton2015) (see also Safi & Pettorelli, Reference Safi and Pettorelli2010; Bland et al., Reference Bland, Orme and Bielby2015) addressed this qualification by extracting as much information as possible out of the existing data using advanced statistical methods. There are a large range of techniques that have been used to infer missing data and it is not possible to review them here, except to point out that suitable methods have been developed (Nakagawa & Freckleton, Reference Nakagawa and Freckleton2008), or that the problem can be dealt with using flexible statistical frameworks, such as Bayesian modelling (Gelman et al., Reference Gelman, Carlin and Stern1995). Another recent application used models to infer the maximal population growth rate of several shark species for which this demographic rate has not been otherwise estimated (e.g. Pardo et al., Reference Pardo, Cooper and Reynolds2018).

In many cases, however, the bottom line is that sufficient data do not exist and there is no option but to collect more. Data are time-consuming and expensive to collect. Engaging in a programme of data collection will delay implementation and use up resources that could be targeted at on-the-ground management. Frequently there will not be resources available for data collection and hence the knowledge gap is never plugged.

On the assumption that more information could be obtained, a key question arises: will collecting more information improve management decisions (Maxwell et al., Reference Maxwell, Rhodes and Runge2015)? Canessa et al. (Reference Canessa, Guillera-Arroita and Lahoz-Monfort2015) highlight a measure called the ‘Value of Information’ (VoI). This measure is the difference in outcome between the expected management action based only on whatever prior information was available, and action taken with new information provided (Yokota & Thompson, Reference Yokota and Thompson2004; Canessa et al., Reference Canessa, Guillera-Arroita and Lahoz-Monfort2015). They provide an example that is typical of many in conservation or land management. Imagine that a species of conservation concern occurs in one location within a protected area. The aim of conservation is to maximise the size of the population in the area over a specified time period. In order to meet this aim, one strategy could be to create a new population. However, imagine further that there is a chance that a disease could be present that would limit the effectiveness of the reintroduction. The VoI in this case reflects the change in estimated effectiveness that would be achieved by testing for the presence of disease before starting the reintroduction programme. Thus, a test might be performed and return a positive or negative result. Given a prior estimate of the prevalence of the disease, the difference between initial and updated estimates can be calculated using Bayesian updating. These differences then measure the VoI provided by conducting testing. This represents the possible improvement in decision-making through the removal of uncertainty.

11.4.3 Addressing uncertainty through benchmarking

A manager might apply a conservation intervention which, if the outcome is positive, leads to a question of whether the intervention should be used again, or even recommended to another manager. Informal communication of outcomes of this sort are not unusual in land management (Henrich, Reference Henrich2001).

From a scientific perspective, this is not an acceptable way of proceeding unless appropriate controls and experimental design are used in the evaluation of the method. Furthermore, the intervention would ideally be evaluated at more than a single site. This reflects, of course, the tension between the situational and general perspectives of practitioners and scientists. There are pitfalls in both views. There is of course, no guarantee that if management appears to work at one site that it is not simply due to natural variation. Figure 11.2a gives an example of this from an agricultural case study. At one site a specific intervention was used and appeared to be successful. However, compared with the outcome on a set of farms that did not use the technique, there is no obviously large effect. On the other hand, if we are too picky about standards of evidence or data then there is a real danger that useful information will be discarded.

Figure 11.2 Uncertainty and benchmarking in weed control. (a,b) Predicted responses of populations of the weed Alopecurus myosuroides to rotational management. The initial frequency of weeds at each sowing density was the same in each case (dashed blue line). Each grey line represents a matrix generated from a different field following two forms of management. (a) What would have been the density (0 = zero, L = low, M = medium, H = high and VH = very high) of an average field had it been planted with spring barley. This is compared with (b) the predicted response from maintaining winter wheat. The red line in (a) represents a single field that was managed with variable sowing densities. Figures (c–e) compare the observed effect of management with difference sources of background variation to disentangle the uncertainty in management. We generated models for each field: 22 in winter wheat and 12 rotated from winter wheat to spring barley, and their results are presented in rank order. The effect range is the estimate of the random effect for each field, location or rotation.

Developments such as evidence-based conservation promote the collation of evidence on the effectiveness of management (Sutherland, Reference Sutherland2003; Sutherland et al., Reference Sutherland, Pullin and Dolman2004; see also Chapter 4). The idea here is twofold. First, if the same management has been used in different places then, even if individual interventions do not meet the criteria of a randomised trial (as in Figure 11.2a), the collective body of evidence might be useful. Resources such as www.conservationevidence.com allow this work to be synthesised. Second, using systematic review approaches, it is possible to synthesise this information to provide answers to management problems (Pullin & Stewart, Reference Pullin and Stewart2006; see also Chapter 7).

In the example shown in Figure 11.2a, a single manager implemented one management intervention. On its own this is not enough to determine effectiveness. However, if many people implement the same management then it may be possible to use non-intervention cases as a benchmark and compare the difference with those places where interventions were made. For example, Figures 11.2a and 11.2b show the distribution of weed population sizes in fields subject to intervention (Figure 11.2a) compared with those in which no intervention was made (Figure 11.2b). There is an apparent difference in outcome, but clearly with a high degree of variance. Modelling the data (Figure 11.2c–11.2e) reveals that, although there is an effect of the intervention (Figure 11.2e), there is also a high degree of variance resulting from the initial state (Figure 11.2c) or from the variation in population dynamics from field to field (Figure 11.2d). Consequently, the effect of management, although measurable (Figure 11.2e), is relatively small compared with the intrinsic variability of this system. In this example, the results in Figure 11.2c–e confirm the expectation that the specific management intervention should work, but they also confirm anecdotal local reports that the effectiveness of this approach is patchy, and suggest that frequently the positive effects observed may be attributable to other factors (the large negative effect sizes in Figure 11.2c and d).

Benchmarking of this sort could be extremely valuable in aiding management decisions (Freckleton et al., Reference Freckleton, Hicks and Comont2018). Technological advances, such as widespread instrumentation of agricultural machinery, UAS technology (Paneque-Gálvez et al., Reference Paneque-Gálvez, McCall and Napoletano2014; Lambert et al., Reference Lambert, Hicks and Childs2018) and remote sensing (Kerr & Ostrovsky, Reference Kerr and Ostrovsky2003; Turner et al., Reference Turner, Spector and Gardiner2003) offer the possibility of widescale automated data collection at massive scales. When combined with ecological models, such data could provide a hitherto impossible resource for reducing uncertainty in predicting future management outcomes.