Impact assessment and snowballs

In the ‘Where are we now?’ section, SSH highlight the many advances that the field has made and state both that “[t]here is much for proponents of behavioural science to be pleased with here” and “…there is a danger that behavioural science is seen to offer merely technocratic tweaks, rather than the more wide-ranging reassessment of public administration that could be possible.” Indeed, the rise in activity, awareness and popularity of the field is exciting, but I would caution that while the “wide-ranging reassessment of public administration” could be possible and worth striving for, it is not likely to be easy.

Looking first at the challenges identified in SSH (the long-term effects of interventions; repeated exposure effects; problems with proxy measures; spillovers and general equilibrium effects and unintended consequences; cultural variation; ‘reverse impact’; and the replication crisis), they seem to have a common theme around impact evaluation. Even randomised controlled trials, often referred to as the gold standard of research, have their limitations, and each of SSH's challenges, with the exception of ‘reverse impact’, illustrates a case where it is difficult to extrapolate the actual sustained impact of a particular intervention from the results of the experimentation preceding it. This highlights both how difficult it can be to accurately measure impact and that even if the pilot is promising, rolling out the scaled intervention will not necessarily have proportional effects.

Looking next to the opportunities SSH highlight as available to the field (influencing the behaviour of government itself; scaling interventions; social diffusion; nudging organisations; and dealing with thorny problems), all seem to be about scope and scale and are somewhat self-referential. For example, the scaling interventions and social diffusion ‘opportunities’ will be required for nudging government, nudging organisations and thorny problems.

By ‘thorny problems’, the authors are describing “trickier, more complex problems.” As above, this opportunity seems to describe all of the areas preceding it within the category; that is, nudging government, scaling interventions, social diffusion and nudging organisations all could themselves be described as thorny by SSH's definition. Further, it may be worth stepping beyond calling these problems ‘thorny’ and instead describing them as ‘wicked’, meaning that they are difficult to solve due to incomplete knowledge, the size of the problem or number of people and stakeholders involved, the complexity and interconnectedness of the system or any combination of those factors (Wicked Problems website; Churchman, Reference Churchman1967).

Perhaps a simpler way to describe the opportunity set, then, is to ask: how can behavioural science help to snowball ‘simple’ individual-level behaviour change out to complex population-level shifts?

These challenges and opportunities are therefore not trivial. They will be about moving beyond the tweaks at the margins and towards broader organisational, systems and social change. This is not to undermine the work of the “low-hanging fruit” of binary choices as described by SSH. Not only does improving tax repayment (Hallsworth et al., Reference Hallsworth, J.A., R.D. and I.2017), for example, contribute to the effective functioning of public services that are so fundamentally important for society, but also, successful behaviour change in these “low-hanging fruit” areas served as proof of concept, and therefore has been instrumentally important for the Behavioural Insights Team (BIT) and the field more generally as well. But what about other, bigger, more complex, wicked issues where it is unlikely that the pull of just one lever will be sufficient for change?

Thorny, wicked problems

SSH allude to wicked interconnectedness when they only briefly discuss the Lucas critique, but this is perhaps the crux of the situation. They write: “…since consumption functions are not fixed and can respond to changes in circumstances, they are not ‘policy invariant’, and so may change in response to changes of policy.” Indeed, this is the essence of what makes it so difficult to measure impact directly. Any effect will necessarily change the system, and therefore we cannot assume that the same intervention will have the same effect again, and given this ever-changing environment, we can never perfectly evaluate the impact of any particular behaviour change initiative.

The World Bank's World Development Report (2015) explains that much of our thinking is done automatically, socially and through the frame of mental models. By ‘automatically’, the World Bank authors are referring to our individual cognition and system 1 processing; ‘socially’ explains the weight of our peer group on our behaviour; and ‘mental models’ refers to how our cultural and institutional experiences shape the lenses through which we view the world. Any change to context could influence any one of those levels, not least our mental models. So when SSH discuss the challenges to the field, it seems that the common element is the complex, ever-shifting context.

For example, SSH ask whether repeated exposure leads to diminishing marginal returns (less impact per additional unit of exposure) or reinforcement (more impact per additional unit of exposure) of the message. This is clearly important to understand, but the general point is once again the same and particularly salient in this case: with every intervention – even if it is the same intervention being repeated – the context changes. Therefore, we cannot know for sure that an intervention that worked in the past will ever work again in the future, because the context is forever new.

A similar point can be made for the social diffusion opportunity. Social diffusion has the potential to increase impact exponentially, clearly improving the efficiency of an intervention. As SSH write, “[i]n a world of limited resources, a better understanding of how to harness peer-to-peer transmission of behaviour could mean that the same or better outcomes are achieved at much less cost…” However, while an intervention may be the catalyst, the social diffusion itself would be largely outside of the policy-maker's control. Whether that behavioural intervention would work similarly again, given the changed context with new norms, would be hard to say. Arguably, the efficiency benefits likely outweigh the uncertainty drawbacks.

With respect to nudging government or organisations, SSH write that organisations are made up of a collection of individual people, each of whom could presumably be nudged, while also noting that there are often policies in place to prevent just one single person from making a large decision. What should be emphasised here is that a whole group is different to the sum of its parts; that is, it cannot be taken as given that the same person would act in the same way alone as when in the presence of others. Therefore, there is no guarantee that an intervention introduced in an organisation would produce the same effect as one in the home of a singleton.

For example, consider the case of defensive decision-making, in which a person chooses a course of action not because it necessarily has the highest expected outcome assuming that all goes well, but because it is more easily defensible in the case that all does not go well (Gigerenzer, Reference Gigerenzer2015). Defensive decision-making may be more prevalent for business or organisational decisions than personal decisions because of the difference in stakeholders and clarity of hierarchy. The consequence of not being able to defend your decision may vary: a friend or neighbour might disapprove of a choice you have made, but your boss can fire you.

Being in a group necessarily changes the social context and can affect the mental models – an organisational culture – through which we make sense of the organisation and the world. As with the discussion of repeated exposure effects, providing nudges in an organisation may result in reinforcing feedback (so that there is positive social contagion), diminishing returns (with some people affected less strongly than others) or even balancing feedback (e.g., if someone were to feel licensing effects off the back of one of their colleagues’ good deeds) within the system, making the ultimate impact harder to predict.

Beyond behavioural interventions

SSH admirably accept the challenge of addressing thorny or wicked problems head on: “[a]t first, this can seem something daunting and to be avoided, but it is precisely here that new tools should be used. If we consider that behavioural economics came into existence to explain phenomena that standard economic analysis found inexplicable, perhaps behavioural science can solve problems that standard economic tools have found insoluble” (emphasis added). Yes, perhaps it can, but it is also worth considering that not all interventions are behavioural, and not all impact is from interventions.

Firstly, standard economic tools should not be discarded. A good behaviour change practitioner will recognise that not all interventions should be behavioural (i.e., nudges) and that the appropriate lever for change may be one of the more traditional approaches (behaviourally informed or otherwise), like awareness or regulation (e.g., see Finighan, Reference Finighan2015; Soman, Reference Soman2015; Thaler, Reference Thaler2017). Dilip Soman (Reference Soman2015) uses personas to illustrate the point: sometimes the best method to change behaviour is that of a lawyer, an economist, a marketer or a behavioural scientist. Note that the behavioural scientist is only one of many in this stylised set-up. Similarly, Finighan (Reference Finighan2015) provides examples of behavioural levers that address standard economic problems and, vice versa, standard policy levers that solve behavioural problems. An example of the former would be to “align teacher interests with student learning by providing social rewards, like recognition and thanks,” and an example of the latter would be “to overcome savings myopia, use a mandate (e.g. superannuation) or a financial incentive to increase savings” (Finighan, Reference Finighan2015, p. 5).

Next, and related, is that the benefit of stating this ambition in the present journal could be amplified by extending the discussion beyond the echo chamber of other behavioural public policy practitioners. There is a lot to learn about the opportunities that SSH identified by reaching out to academics and even commercial organisations. For better or for worse, we can be sure that giants like Facebook are researching social diffusion, and there could be much for policy-makers to learn from those insights as they are exploring ‘network nudges’ (Chambers, Reference Chambers2014). Looking towards the newest technology, others are exploring how machine learning could identify different subgroups of people to facilitate targeted and personalised interventions, mitigating the risk of unintended consequences from blanket nudges that do not work on everyone in the population (Risdon, Reference Risdon2017).

Once armed with a full suite of tools, both standard and behavioural, and learning from a range of forward-looking sources, a potentially useful way to approach behaviour change given the limitations on impact assessment discussed above is to follow the Royal Society for the Encouragement of Arts, Manufacture and Commerce's advice to “think like a system, act like an entrepreneur.” What they mean by thinking like a system is to try to understand not only the individual person-level motivations for a given behaviour, but also the social and hierarchical pressures, all while recognising that systems are complex and messy. Acting like an entrepreneur is to appreciate that one will never be able to fully understand or measure everything, but nevertheless should seek opportunities to create change where needed rather than preserving the status quo (Burbidge, Reference Burbidge2017). Although we cannot be completely certain of the isolated impact of each intervention, there is some expectation that small wins accumulate towards making a meaningful dent in wicked problems.

And not all impact is from interventions. Behavioural science, and psychology more generally, is extremely useful for giving people a shared language to talk about common experiences and tendencies. Perhaps this is not directly relevant to shaping policy itself, but it is potentially valuable for some of the opportunities identified by SSH, such as when trying to nudge organisations or government. For example, speaking with colleagues about the ‘planning fallacy’ or ‘present bias’ may help grease a conversation about project timelines and what types of ‘commitment devices’ may be useful for staying on track; it may make the difference between the colleague feeling personally attacked for their self-perceived character flaws and feeling like this is support needed for any ‘normal’ human being.

Clear skies?

When describing the origins of BIT, SSH write about their guiding principles and strategy. These seem to be right and admirable intentions and a pragmatic approach. However, there is still an awkward and open question around the mantra ‘nudge for good’. It is certainly better than ‘nudge for bad’, but, even with good intentions, does the choice architect always know what is in someone else's best interest? What may be good for many might be bad for others. How does one avoid harming a group, especially the most vulnerable among us?Footnote ¹ The authors themselves point out that one of the challenges of behavioural public policy is unintended consequences. What happens when an improvement in one outcome measure results in a degradation of another? Or when an intervention results in someone being objectively better off (e.g., by having more saved for retirement) but subjectively worse off (e.g., by increased anxiety juggling lower finances day to day), or vice versa?

These issues of paternalism, population heterogeneity and trade-offs are not unique to behavioural public policy, and are important questions for any policy-maker or any behavioural science practitioner to reflect upon. However, these questions are perhaps especially important for government and other institutional bodies, where issues of both choice and trust come into play. An informed consumer may choose to avoid buying goods or services from a particular company, but one cannot (easily) choose to opt out of participating in civic society.Footnote ² Therefore, ethical considerations should always be top of the mind because while ‘nudge for good’ can and should be the North Star, the sky is not always cloudless, however good the intentions.