2.1 What Do You Mean by ‘Cause’?

Experiments are often, and for good reasons, portrayed as a gold standard to establish causality. But what does it mean exactly that X causes Y? How can we find out that it does? Perhaps Y causes X. Perhaps it goes both ways. Perhaps another variable affects both. Perhaps it is mere coincidence. The ontological and epistemological issues of causality have long been discussed; see, for example, Brady (Reference Brady and Robert2011) or Thye (Reference Thye, Webster and Sell2014) for a user-friendly introduction for social scientists, Mahoney and Acosta (Reference Mahoney and Acosta2021) and Woodward (Reference Woodward and Edward2016) for a more in-depth review of ‘causality as regularity’ and ‘causality as manipulation’ types of theories, respectively. Classic books include Spirtes (Reference Spirtes, Glymour and Scheines2001) and Pearl (Reference Pearl2009). Here, we report a small part of this discussion which seems to be most relevant for the design of experiments in social sciences.

The pioneer of the investigation of causality, David Hume, famously chose the game of billiards to illustrate it: ‘Here is a billiard ball lying on the table, and another ball moving toward it with rapidity. They strike; and the ball which was formerly at rest now acquires a motion. This is as perfect an instance of the relation of cause and effect as any which we know, either by sensation or reflection.’ Indeed, the collision, with no contributing or intermediate factors involved, will always cause the second ball to start rolling. Moreover, the physical mechanism is well understood; we also can easily establish that the second ball remains motionless if nothing strikes it. Social sciences tend to provide us with less-than-perfect instances of cause and effect. Social phenomena tend to be complex and probabilistic and hardly ever have a single cause. In fact, ‘everything is related to everything else’. Moreover, the laws proposed to explain social phenomena are often contested and their validity may vary over time and across cultures. Still, the basic principles of defining and identifying causality carry over from the simpler, physical phenomena.

2.2 Counterfactuals and Randomisation Bias

Suppose we have $p$ treatments (one of which may be the ‘control’, i.e. no treatment); the treatments are labelled $1,\dots ,p$, we give treatment $j$ to experimental unit $i$, and denote the outcome by $Y_i\left(j\right)$. In our experiment, we obtain the result of applying treatment $j$ to unit $i$, but we would like to infer what the outcome would have been if any of the other treatments had been given; in other words, when we observe $Y_i\left(j\right)$, we would like to infer the values of $Y_i\left(k\right)$ for all $k\ne j$. This is termed counterfactual, since for run $i$ we gave treatment $j$ and we did not give any of the other treatments.

We can represent the situation by a causal diagram (Figure 1); the outcome $Y$ is influenced by the treatment $T$ and other causes $C$, which may influence which treatment is given as well as having a direct influence on the outcome.

Figure 1 Common cause, treatment, outcome.

The treatment variable $T$ takes values in $\{1,\dots ,p\}$. We would like to infer that $Y_i\left(j\right)$ (the so-called potential outcome for subject $i$ under treatment $j$ – the outcome that would have occurred had subject $i$ been administered treatment $j$) is independent of $T$ (the treatment actually given), which implies that $E\left[Y_i\right(j)|T=k]$ does not depend on $k$ (and hence that, for each $j$ and $k$, $E\left[Y_i\right(j)|T=k]=E\left[Y_i\right(j\left)\right]$). This is (of course) true if treatments are assigned at random and a proper controlled experiment is carried out (i.e. the sample is representative of the population at large and treatments are randomly assigned). This would break the causal link between the common causes $C$ and treatment $T$ and result in the causal diagram of Figure 2 (since $T$ is enforced by the experimenter, we denote this by a square node).

Figure 2 Intervention breaks the link between common cause and treatment.

Randomisation of treatment means that the choice of treatment is applied irrespective of the values of any hidden covariates that may have a causal effect on both treatment and outcome.

In many situations, though, there are serious obstacles to the construction of a controlled experiment. For example, a sample drawn from college students who agree to participate in an experiment may differ in important ways from those who choose not to participate, so the whole sample may be biased. In experimental economics, an important reason for differences is related to the structure of pay-offs. Harrison et al. (Reference Harrison, Lau and Elisabet Rutström2009) hypothesise that the anticipated variance of pay-offs may affect the profile of risk attitudes of the sample of people who sign up to take part. They confirm that announcing a guaranteed show-up fee leads to a relatively risk-averse sample. Therefore, the problem is not whether treatments are assigned in a suitable random manner to a pool of candidates constituting a representative random sample; the whole sample of possible candidates may be biased with respect to the prevailing characteristics of the population.

Random assignment may discourage some participants. Levitt and List (Reference Levitt and List2009) point out that in clinical drug trials, it seems much harder to persuade patients to participate in a randomised controlled experiment than to persuade them to take a new drug in a non-randomised study. The sample will therefore tend to be skewed for randomised controlled trials.

Framing might matter as well. As pointed out by List (Reference List2021), in field experiments, the use of the word ‘experiment’ itself can cause difficulties, while terminology such as ‘trials’ and ‘pilot studies’ may be more acceptable. This applies not only to participants, but also to non-academic partners who are often necessary to run a field experiment. Some of them are prone to respond along the lines of ‘we’ve been in this business for thirty years, we know what to do, and you’re telling us to choose something at random?!’

2.3 Randomisation Procedures

At this point, we introduce another important ingredient, which is how to assign treatments to subjects who have agreed to participate. Many procedures have been proposed for the random assignment of participants to treatment groups. We outline common randomisation techniques, including simple randomisation, block randomisation, stratified randomisation, and covariate adaptive randomisation. We describe the methods, giving advantages and disadvantages.

2.4 Gosset versus Fisher, or Is Artificial Randomisation Really Necessary?

In some settings, comparability of the treatments may be achieved without explicit randomisation. This goes back to balanced designs proposed by Gosset to study the effect of manure on crop yields. The challenge was that the plots into which the field could be divided were not homogeneous. Gosset’s contribution is discussed by Ziliak (Reference Ziliak2014), who challenged some of the findings of Levitt and List (Reference Levitt and List2009). While we prefer to sidestep any controversy, we would like to draw attention to some important conclusions of Ziliak. He compares Gosset’s balanced designs to random designs; for Gosset’s experiments studying the effects on crop growth of adding manure, the balanced design demonstrated greater power and efficiency than the randomised design.

In that situation, the field was subdivided into plots. On some plots, manure was added, while other plots were left without manure. The plots were not homogeneous, since the field had ridges of fertility. Letting $A$ denote control and $B$ – treatment (which indicated that manure was added), Gosset arranged the control / treatments pairwise, so that the design appeared as: where the ABBA ... layout is in the direction of increasing fertility. The idea was to then take the results pairwise and consider the difference in crop yields between a control plot A and the adjacent treatment plot B.

When there is a positive correlation between adjacent plots, this model presents greater power for estimating the difference in average crop yield between treatment and control. The increased efficiency obtained by using the balanced design can be seen from the following back-of-envelope calculation. Suppose that each observation is the outcome of a normal random variable. Let $X_{A,i}-X_{B,i}$ denote the difference in treatments for the pair labelled $i$ and there are $n$ pairs. Then $\text{Var}(X_{A,i}-X_{B,i})=2{\sigma }^2(1-\rho )$ where ${\sigma }^2$ is the variance; $X_{A,i}\sim N({\mu }_A,{\sigma }^2)$ and $X_{B,i}\sim N({\mu }_B,{\sigma }^2)$ and $\rho =\text{Corr}(X,Y)$ where $X$ and $Y$ denote results for adjacent plots. Then, if we have $n$ adjacent pairs, which we may assume independent,

$\frac{\sum _{i=1}^n\left(\right(X_{A,i}-X_{B,i})-({\mu }_A-{\mu }_B\left)\right)}{\sqrt{n}\sqrt{2}\sigma \sqrt{1-\rho }}\sim N(0,1).$

Let $s^2=\frac{1}{n-1}\sum _{i=1}^n\left(\right(X_{A,i}-X_{B,i})-({\bar{X}}_{A,.}-{\bar{X}}_{B,.}){)}^2$, then

$\frac{\sqrt{n}\left(\right(({\bar{X}}_{A,.}-{\bar{X}}_{B,.}))-({\mu }_A-{\mu }_B\left)\right)}{s}\sim t_{n-1}$

at least approximately. Since $E\left[s^2\right]=2{\sigma }^2(1-\rho )$, we can see that, taking the observations pairwise and observing the differences, the balanced design is more efficient and powerful than a randomised design when $\rho$ is substantially larger than $0$.

Clearly, if the experimental units are known not to be homogeneous and the deviation from homogeneity is understood (in Gosset’s example, he knew where the ridges of fertility were), then the balanced design enables the confounding effect of fertility ridges to be removed more efficiently than artificial randomisation; we see serious disadvantages to the naive application of artificial randomisation.

The natural analogy in economics is the flow of subjects logging in one by one to an online experiment. Subjects who log in at approximately the same time may have similar characteristics (they come from the same time zone, they have the same employment status, maybe they belong to the same tutorial group, just after class has finished). It may therefore be advisable to assign them to treatments A and B according to the ABBAABBA pattern of Gosset, rather than randomly. This is of special importance when there is some clustering (and the whole group has to be assigned to one treatment).

When the experiment is performed in person, or with online human assistance (so that there is some interaction between the subject and the experimenter), the disadvantage of such predetermined treatment assignment is that it does not allow the experiment to be double-blind. The experimenter may thus inadvertently affect subjects’ behaviour. A solution that introduces uncertainty about the next subject’s treatment assignment but (almost) guarantees (near) balance is that of biased coin randomisation Smith (Reference Smith and Balakrishnan2014). Hitherto applied in medical trials, it involves treatment assignment which is random, but not independent of previous assignments; the treatment in which the current number of observations is lower has a greater chance to be assigned to the next subject.

2.5 Within-Subject and Between-Subject

The issue of whether to use within-subject or between-subject (or hybrid) designs pervades the design of experiments for economics in a fundamental way. The question is whether to use the same subject for repeated runs of the experiment in different treatments (within subject) or to use different subjects for different treatments (between subjects). While within-subject experiments can be more economical, there are disadvantages to using the same subject for several experimental runs; the subject typically has a memory of the previous ones and so repeated runs with the same subject cannot be as independent draws from the same (subject-specific) distribution (The errors will clearly not necessarily be independent and identically distributed (IID) if the subject remembers the previous responses. For example, if the subject is asked exactly the same question twice, there may be a random element in the first response; the subject is likely to give exactly the same response the second time through, without thinking again.)

Pricing a Ham Sandwich

Charness, Gneezy, and Kuhn (Reference Gary, Gneezy and Kuhn2012) use the following illustrative example in their discussion of relative merits of within-subject and between-subject experiments. A researcher may ask participants how much they would be willing to pay (WTP) for a sandwich:

1. (a) if they were to buy a sandwich from their neighbourhood bakery, and

2. (b) if they were to buy it at the airport.

Assuming that the sandwich is identical in both situations and so is actual and anticipated hunger, there is little reason for the WTPs to be any different. Yet people seem to end up willingly overpaying badly at airports, but not in bakeries. Is it just because these are different people or because the food is correctly anticipated to be even worse and pricier during the flight? Or does this discrepancy reveal something about our flawed judgement, whereby € 8.95 for a sandwich only seems reasonable in one situation but not in the other?

We can try to disentangle these competing explanations in a controlled study. When doing so, we must decide whether the same person answers both questions (within-subject design) or each person only answers one question (between-subject design), one half of the sample being randomly assigned to the question of how much they would pay at the bakery and the other half to the question of what they would pay at the airport. We first discuss some advantages of the within-subject design compared to the between-subject approach and then the other way around.

Advantages of the Within-Subject Design

Within-subject design has quite a few important advantages over between-subject design. Firstly, the validity of the analysis does not depend on random assignment; both treatments are assigned to the same individual. Obviously, comparing WTPs elicited at an airport to those elicited at a bakery would be worthless, since air travellers are not a random sample of bakery shoppers (or vice versa). A less obvious case is when there is non-trivial and treatment-specific dropout. In contrast to a between-subject design in a within-subject experiment, we can easily restrict our analysis to those who actually made choices in both treatments, thereby distinguishing pure treatment effects from selection effects.

Secondly, for the same reason, within-subject designs can offer a significant increase in statistical power, because there is no noise associated with individual differences. Naturally, as long as the assignment of the treatment is random, causal estimates can be obtained in the between-subject design by comparing the differences in response of the subjects between the various experimental conditions, but the between-subject noise may be substantial. This means that the sample has to be large enough to account for this; if only the sample average is required and ${\sigma }^2$ denotes the between-subject variance, then the sample average variance is $\frac{{\sigma }^2}{n}$ and $n$ has to be sufficiently large to make this sufficiently small.

For example, suppose we are considering the difference between two treatments and we are in a situation where we can use the same subject for two experimental runs, one for each treatment, and suppose the outcome when treatment $j$ for $j\in \{1,2\}$ applied to individual $i$ is modelled as:

$Y_{i,j}=\mu +{\alpha }_j+{ϵ}_{i,j},$

where $\left(\begin{array}{c}{ϵ}_{i,1}\\ {ϵ}_{i,2}\end{array}\right)\sim N\left(\left(\begin{array}{c}0\\ 0\end{array}\right),{\sigma }^2\left(\begin{array}{cc}1& \rho \\ \rho & 1\end{array}\right)\right)$ and $({ϵ}_{i1},{ϵ}_{i2}{)}^{\prime }$ are uncorrelated for different subjects. This could arise, for example, if we are considering two different medications for reducing blood pressure, where a subject may have underlying health conditions which affect all treatments in such a way that the within-subject errors are correlated. For such a model, with $n$ subjects $i=1,\dots ,n$

${\hat{\alpha }}_2-{\hat{\alpha }}_1={\bar{Y}}_{.2}-{\bar{Y}}_{.1}\sim N\left({\alpha }_2-{\alpha }_1,\frac{2{\sigma }^2(1-\rho )}{n}\right).$

To get a variance of $\frac{2{\sigma }^2(1-\rho )}{n}$ from a between-subject experiment, where each subject is used exactly once, we would require $N_B=\frac{2}{1-\rho }{N}_W$ subjects.

We can see that the reduction in number for an experiment with the same precision (to carry out statistical tests with the same power) is not simply from the fact that the subjects in a between-subject are only used once, but also due to the correlation of the random effects within subjects.

Thirdly, within-subject designs tend to be more efficient in terms of data collection. Typically, the two treatments share a substantial part of the instructions and stimuli. Thus, a decision in Treatment B may be elicited more quickly and easily from a subject who has already made a decision in Treatment A (the subject only has to understand how the two are different) than from a fresh subject, as would have been the case in the between-subject design. This is not an issue in the sandwich example, but it certainly is with more extensive instructions.

Fourthly, within-subject experiments may be more appealing theoretically. For example, if we have a theory predicting the effect of price, we may want to observe actual price change and therefore expose the same agent (rather than two different agents) to two different prices. Likewise, when investigating preference reversals (Tversky and Thaler (Reference Amos and Thaler1990)), we want to observe different choices of the same individuals. The underlying theory often takes the form of some individual utility function. Within-subject designs allow this to be estimated, while between-subject designs generally do not.

Fifthly, within-subject designs yield individual-specific estimates of the treatment effect. This is important if we expect heterogeneity, especially if we want to link these effects with some other behavioural characteristics. Further, within-subject design is the only option if we want to select subjects based on a given trait. Suppose, for example, that we want to know if markets populated with individuals prone to preference reversals are more erratic. The most effective way to find out will be to measure preference reversals in a within-subject design, merge the subjects into groups based on their responses, and then compare the behaviour of high-reversal groups to that of low-reversal groups.

3.1 Introduction

We aim to find a design $X$ from the space of possible designs $X$, which minimises some functional $F$, namely the optimal design is ${\text{argmin}}_{X\in X}F\left(X\right)$. The choice of $F$ depends on the sense in which the design is optimal; which criterion do we choose for optimality? It also depends on the computational power available to solve the minimisation problem.

In straightforward settings (e.g. the classical standard factorial designs – one way model, Latin square, $2^k$ factorial, and fractional factorial designs), there are well-motivated standard designs available where the motivation is intuitive and the designs facilitate parameter estimation and hence prediction; they represent good ‘all-around’ designs, although they may not be optimal with respect to any particular criterion that an experimenter has in mind. When the situation becomes more involved, these designs may not be feasible.

In Section 4, we discuss DCEs, where it is not possible to consider all treatment combinations. For example, suppose we are interested in estimating how choice is affected by six factors (or attributes), each coming at three different levels. There are $3^6=729$ possible treatment combinations. Suppose we observe binary choices between these; there are $\frac{1}{2}\times 729\times 728=\mathrm{265,356}$ different pairs. Few respondents will be willing to sit long enough to answer $\mathrm{265,356}$ questions. Moreoever, many of these comparisons will be ‘no-brainers’ (or, more formally, cases of weak dominance). If we know that Option A is better than Option B on some attributes and not any different (taking the same levels) on remaining attributes, then asking a respondent to choose from them is mostly interesting as a way to check if they are still awake after so many choices.

Only a very small fraction of possible choice sets is thus presented and the question, of course, is which they should be. A natural idea is to randomise levels independently. This approach guarantees (with sufficient data) that we can identify the impact of each attribute, but it is not necessarily an optimal approach. In particular, it does not exclude comparisons involving dominance, from which we do not learn much. How do we measure how much we expect to learn?

When designing an experiment, we have to make several decisions before we collect data. These decisions are: which covariates are to be included in the model (and hence the number of unknown parameters) and the values of those covariates over which the experimenter has control. The key elements of the statistical model are as follows:

• A response variable $Y$ (or vector $Y$).

• A design of covariates $X$ that the experimenter chooses.

• A parameter $\theta$ (or parameter vector $\theta$) of interest. We split $\theta$ into two parts, $\theta =(\beta ,\sigma )$ where $\beta$ denotes the model parameters (how the covariates influence the outcome) and $\sigma$ the ‘noise’ (or dispersion) parameters. In a standard Gaussian model, $\sigma =\sigma$, a single parameter, which is the standard deviation of the error. We choose the length of $\beta$ when we decide on the model, but the parameters themselves are unknown and the aim of the design problem is to choose an experiment which gives as much information as possible about them.

Note: the matrix $X$ may contain columns where the values are not chosen by the experimenter. For example, consider the Gaussian linear model $Y=X\beta +ϵ$ where $Y$ is an $n$-vector of observations, $ϵ\sim N(0,{\sigma }^2{I}_n)$. Suppose there is an intercept parameter ${\beta }_0$, so that

$Y_i={\beta }_0+\sum _{j=1}^p{x}_{ij}{\beta }_j+{ϵ}_i.$

There are $p$ covariates; the value of covariate $j$ for run $i$ is $x_{ij}$. The design matrix may be written as $X=\left({\text{1}}_n|\tilde{X}\right)$ where ${\text{1}}_n$ denotes an $n$-vector with each element $1$ and $\tilde{X}$ is the $n\times p$ design matrix with entries ${\tilde{X}}_{ij}=x_{ij}$. The mean value space $M=\{X\beta :\beta \in R^{p+1}\}$ is a $p+1$ dimensional subspace of $R^n$ and the parameter vector $\beta$ (which includes the intercept ${\beta }_0$) is a $p+1$ vector.

The experiment is designed to provide: (a) an estimator $\hat{\theta }=f(y,X)$ of $\theta$, which is a function of the response and covariates and (b) the distribution of the estimator, which we denote by $\pi (\theta |y,X)$. In the classical setting, $\hat{\theta }$ may be (for example) an ordinary least squares (OLS) estimator or maximum likelihood estimator, while in the Bayesian setting, $\pi (\theta |y,X)$ is the posterior distribution and we use $\hat{\theta }$ to denote a random vector with this distribution. With prior $\pi \left(\theta \right)$ and data likelihood $p(y|X,\theta )$ (the probability density for outcome $y$, when an experiment with design $X$ is carried out and $\theta$ is the true parameter vector), the posterior distribution (by Bayes rule) is:

$\pi (\theta |y,X)=\frac{\pi \left(\theta \right)p(y|\theta ,X)}{\int \pi \left(\theta \right)p(y|\theta ,X)d\theta }.$

For experimental design, the decision comes in two parts; before data is collected, a design (say $X$) is chosen from a set of possible designs, say $X$. An experiment is then run according to the design and data $y$ is collected, where $y\in Y$; $Y$ denotes the sample space. Based on the data, a conclusion $d$ is reached, from a space $D$ of possible conclusions (e.g. from $p$ possible treatments, a decision as to whether one of them is significantly better than the others and, if so, which is best).

In the classical setting, optimality is framed in terms of loss functions and risk. Let be the data and $l(\theta ,f(y,X);X)$ the loss incurred by making a decision $f(y;X)$ when the true parameter vector is $\theta$, using design $X$; the design $X$ and estimator $f(Y;X)$ chosen to minimise the risk; the true parameter value $\theta$ is, a priori, unknown, where the risk for $f$ is defined as:

$R(\theta ,f;X)=E_{\theta }\left[l\right(\theta ,f(Y;X);X\left)\right].$(3.1)

When parameters $\beta$ are in view, a standard choice for $f(Y;X)$ would be the maximum likelihood estimator ${\hat{\beta }}_{ML}$ and, for the loss, $l(\theta ,f(Y;X);X)=|\beta -{\hat{\beta }}_{ML}{|}^2$ (squared error loss). Then:

$R(\theta ,f;X)=E_{(\beta ,{\sigma }^2)}[|{\hat{\beta }}_{ML}-\beta {|}^2].$

Formulating it as a Bayesian decision problem, we construct a utility function $u(\theta ,X,y)$, representing the utility when data $y$ is obtained, $\theta$ is the parameter vector, and $X$ is the design chosen. For a design $X$, the expected utility is given by:

$U\left(X\right)={\int }_Y{\int }_{\Theta }u(\theta ,X,y)p(y|\theta ,X)\pi \left(\theta \right)d\theta dy,$(3.2)

and the optimal design is $X^{*}={\text{argmax}}_{X}U\left(X\right)$ (the design that maximises $U\left(X\right)$). Here $p(y|\theta ,X)$ is the density for $y$ given $\theta$ and $X$ and $\pi \left(\theta \right)$ is the prior density for $\theta$.

We discuss various optimality criteria, so-called ‘alphabet’-optimality (both classical and Bayesian) and also the SIG criterion introduced by Lindley (Reference Lindley1956) for optimal design. The SIG criterion is of particular importance in the discussion of adaptive designs (Section 5), since it is based on consistency between splitting an experiment into several components and treating the experiment as a whole.

In general, we aim to estimate quantities which are functions of model parameters $\beta$. For example, when studying WTP and labelling ${\beta }_0$ the coefficient of price and ${\beta }_1,\dots ,{\beta }_p$ the non-price attributes, we are interested in the quantity ${\gamma }_i:=-\frac{{\beta }_i}{{\beta }_0}$, which measures the respondent’s WTP per unit cost for non-price attribute $i$.

Having decided on $X$, we gather data $y=(y_1,\dots ,y_n{)}^{\prime }$. The Bayesian optimal design is the design which maximises the expected utility (3.2). Parameter estimation is an important objective. It is also a key step on the way to computing a predictive distribution, which predicts, with as much certainty as possible, where future observations lie.

In optimal design theory, the optimality (or design) criteria are often direct functions of the information matrix, chosen to optimise accuracy either for parameter estimates or prediction. Except for the simplest cases (e.g. Gaussian), the information matrix depends on the unknown parameters to be estimated. The Bayesian approach deals with this by placing a prior distribution over the parameters. In general, Bayesian optimal designs outperform locally optimal designs based on a single prior choice of parameter. It is intuitively clear why this should be the case, but the enhanced performance comes at substantial additional computational cost for calculating the design.

The algorithms for finding (at least approximately) an optimal design according to the criteria discussed in what follows and their implementation may be found in the acebayes package for R; see Overstall and Woods (Reference Overstall and Woods2017) for a comprehensive treatment of the derivation of the algorithms and their use.

3.2 Alphabet Optimality

Some of the most popular critera are the so-called alphabet-optimality criteria. They can broadly be divided into two categories, those based on parameter estimation and those based on the predictive distribution.

Alphabet-optimality critera arose with the Gaussian linear model $Y=X\beta +ϵ$ in view, where $ϵ\sim N(0,{\sigma }^2{I}_n)$, with respect to the performance of the OLS estimator ${\hat{\beta }}_{\text{OLS}}=(X^{\prime }X{)}^{-1}{X}^{\prime }Y\sim N(\beta ,{\sigma }^2\left(X^{\prime }X{)}^{-1}\right)$. For this situation, the estimator ${\hat{\beta }}_{\text{OLS}}$ is unbiased and BLUE (best linear unbiased estimator) in the sense that, among unbiased estimators, this is the estimator that minimises $\text{Var}(\sum c_j{\hat{\beta }}_j)$ for any linear combination $\sum _j{c}_j{\beta }_j$ of the parameters. One important feature here is that $\text{Cov}\left({\hat{\beta }}_{\text{OLS}}\right)={\sigma }^2(X^{\prime }X{)}^{-1}$ does not depend on $\beta$ (the true value of the parameter) and the ‘optimal’ design will not depend on ${\sigma }^2$. Discrete choice experiments (where each response is a choice of a particular option from two or more) do not fall into this framework.Criteria Based on Parameter Estimation

• D-optimality The most popular optimality criterion to design choice experiments is the D-optimality criterion. The D-optimality criterion seeks to maximise the determinant of the information matrix (equivalently, it minimises the determinant of the inverse).

  For the Gaussian linear model $Y=X\beta +ϵ$, $ϵ\sim N(\text{0},{\sigma }^2{I}_n)$, we have

  $p\left(y\right|\beta ,{\sigma }^2,X)=\frac{1}{{\left(2\pi \right)}^{n/2}{\sigma }^n}\exp \left\{-\frac{1}{2{\sigma }^2}(Y-X\beta {)}^{\prime }(Y-X\beta )\right\},$
  so that:
  $I\left(\beta \right)=-{\nabla }_{\beta }{\nabla }_{\beta }logp(y|\beta ,{\sigma }^2,X)=\frac{1}{{\sigma }^2}\left(X^{\prime }X\right).$
  A D-optimal design therefore maximises $\text{det}\left(I\right(\beta \left)\right)$; equivalently, it minimises $\text{det}\left(\right(X^{\prime }X{)}^{-1})$. Since $\text{Cov}\left({\hat{\beta }}_{\text{OLS}}\right)={\sigma }^2(X^{\prime }X{)}^{-1}$, this is equivalent to finding the design which minimises $\text{det}\left(\text{Cov}\right({\hat{\beta }}_{\text{OLS}}\left)\right)$.

  Let us turn to the Bayesian setting, with prior $\pi (\beta ,{\sigma }^2)={\pi }_1\left(\beta |{\sigma }^2\right){\pi }_2\left({\sigma }^2\right)$, where ${\pi }_1\sim N({\beta }_0,{\sigma }^2{\Omega }_0^{-1})$ (using ${\beta }_0$ to denote the prior mean). Usually ${\pi }_2\sim \text{Scale-Inv}{\chi }^2(\nu ,{\tau }^2)$ (the scale inverse chi squared is the standard conjugate prior for an unknown variance), although the choice of ${\pi }_2$ does not affect the design and can therefore be safely omitted from the discussion. We let

  $J\left(\beta \right):=-{\nabla }_{\beta }{\nabla }_{\beta }log\pi (\beta ,{\sigma }^2)=\frac{1}{{\sigma }^2}({\Omega }_0+(X^{\prime }X\left)\right),$
  where ${\nabla }_{\beta }{\nabla }_{\beta }$ denotes taking the matrix of second derivatives. This is the Bayesian counterpart of the Fisher information matrix for the parameter vector $\beta$ (the prior has been incorporated). We consider a utility function $u(\theta ,X):=\text{det}\left(J\right(\beta \left)\right)$ and take a design that minimises $U\left(X\right)$ (defined by (3.2)). The Bayesian D-optimal design defined in this way maximises $\text{det}({\Omega }_0+(X^{\prime }X\left)\right)$ (or, equivalently, minimises $\text{det}\left(\right({\Omega }_0+\left(X^{\prime }X\right){)}^{-1})$.

  More generally, in the framework of maximising expected utility (3.2), we take the utility function for design $X$ as the determinant of the Fisher information for the parameters of interest:

  $u(\theta ,X)=-\text{det}\left(E_{\theta }\left({\nabla }_{\beta }{\nabla }_{\beta }log\pi (\theta |X,Y)\right)\right).$(3.3)
  where the expectation $E_{\theta }$ is with respect to $p(y|\theta ,X)$ (the utility therefore does not depend on $y$).

  For example, consider the logistic regression model.

  Logistic Models Logistic models fall under the umbrella of generalised linear models and we refer to Agresti (Reference Agresti2012) for a good treatment of this subject. The logistic model, in its simplest case, is a model for binary data, where $Y\sim \text{Be}\left(\pi \right)$ (Bernoulli trial) and the success probability $\pi$ depends on covariates $x=(x_1,\dots ,x_p)\in R^p$. The model is said to be logistic if the logit function of the success probability for covariate values $x=(x_1,\dots ,x_p)$ may be written as

  $log\frac{\pi \left(x\right)}{1-\pi \left(x\right)}=\sum _{j=1}^p{x}_j{\beta }_j$(3.4)

  for a parameter vector $\beta =({\beta }_1,\dots ,{\beta }_p{)}^{\prime }$. This is the model referred to where the utility (3.5) is suggested for a Bayesian D-optimal design. Other standard models for binary data are generated by taking $\pi \left(x\right)=\Phi \left(\sum _j{x}_j{\beta }_j\right)$ where $\Phi$ is the cumulative distribution function (CDF) of a random variable. Two important examples generated in this way are:

  1. – The probit model, where $\Phi \left(z\right)={\int }_{-\infty }^z\frac{1}{\sqrt{2\pi }}exp\left(-\frac{y^2}{2}\right)dy$ (the standard normal CDF) and

  2. – The Gumbel or extreme value model, where $\Phi \left(z\right)=exp\left(-exp\left(z\right)\right)$, so that

     $log\left(-log\pi \left(\sum _j{x}_j{\beta }_j\right)\right)=\sum _j{x}_j{\beta }_j.$

  The optimal design depends strongly on the model specification; a design that is optimal for model (3.4) will not necessarily be optimal for a probit or extreme value model.

  For the logistic regression model, we have $Y_1,\dots ,Y_n$ independent Bernoulli trias, where $Y_i\sim \text{Be}\left(\pi \right(x_i\left)\right)$, $x_i=(x_{i1},\dots ,x_{ip}{)}^{\prime }$ denotes the covariate values for $Y_i$. We therefore have:

  $p(y|\beta ,X)=exp\left(\sum _{i=1}^n\sum _{j=1}^p{y}_i{x}_{ij}{\beta }_j+\sum _{i=1}^{n}log(1-{\pi }_i(x_{i.}\left)\right)\right).$
  A standard choice of prior is to take independence of the parameters under the prior and ${\beta }_j\sim N({\beta }_{j,0},s_j^2)$ for $j=1,\dots ,p$. and (3.3) becomes:
  $u(\beta ,X)=\text{det}\left(X^{\prime }W\left(\beta \right)X+\text{Diag}\left(\frac{1}{s_1^2},\dots ,\frac{1}{s_p^2}\right)\right),$(3.5)

  where $W\left(\beta \right)=\text{Diag}\left({\pi }_1\right(1-{\pi }_1),\dots ,{\pi }_n(1-{\pi }_n\left)\right)$. The first part is from the Fisher information and the second part is from the prior. The important point to note is that for logistic regression, unlike the Gaussian case, the Fisher information depends on $\beta$. With prior $\pi \left(\beta \right)$, the utility to be maximised is $U\left(X\right)$ from (3.2).

  For optimal estimates of a vector $\gamma \left(\theta \right)$ of length $p$, the Bayesian D-criterion value, known as the $D_B$- criterion value, is:

  $D_B={\int }_{\Theta }{\left(\text{det}\left(I^{-1}(X,\gamma (\theta \left)\right)\right)\right)}^{1/p}\pi \left(\theta \right)d\theta ,$(3.6)
  where the exponent $1/p$ is inserted to ensure that the value is independent of the dimension $p$ of the vector $\gamma$. Minimising this value results in the $D_B$-optimal design.

  $D_s$ optimality is a variant of $D$-optimality, where we base the utility on a relevant subset of the parameters and minimise the determinant of the covariance matrix of this subset.

• A-optimality

  An A-optimal design (classical setting) is a design which minimises the trace of the inverse of the information matrix. (For the Gaussian linear model, an A-optimal design is one which minimises $\text{Trace}\left(\right(X^{\prime }X{)}^{-1})$, which is equivalent to the design which minimises $\text{Tr}\left(\text{Cov}\right(\hat{\beta }\left)\right)$, where $\hat{\beta }=(X^{\prime }X{)}^{-1}{X}^{\prime }Y$ is BLUE).

  For Bayesian A-optimality (general setting), let ${\beta }_{\text{MEP}}=\int \beta \pi (\theta |y,X)d\theta$ denote the mean posterior estimate of $\beta$); then a Bayesian A-optimal design is a design which minimises $U\left(X\right)$ of (3.2) with the negative squared error loss utility function:

  $u_{\text{NSEL}}(\beta ,X)=-(\beta -{\beta }_{\text{MEP}}{)}^{\prime }(\beta -{\beta }_{\text{MEP}}).$(3.7)
  Note that, for the Gaussian linear model with prior $\pi (\beta ,{\sigma }^2)={\pi }_1\left(\beta |{\sigma }^2\right){\pi }_2\left({\sigma }^2\right)$, ${\pi }_1\left(\beta |{\sigma }^2\right)\sim N({\beta }_0,{\sigma }^2{\Omega }_0^{-1})$ (  ${\beta }_0$ denotes prior mean) and ${\pi }_2\left({\sigma }^2\right)$ is the prior over the variance parameter ${\sigma }^2$, we have
  ${\beta }_{\text{MEP}}=\int \beta {\pi }_1\left(\beta |{\sigma }^2\right)d\beta ;$
  ${\beta }_{\text{MEP}}$ does not depend on $\sigma$. the design is A-optimal if and only if it minimises $\text{Trace}\left(\right({\Omega }_0+\left(X^{\prime }X\right){)}^{-1})$ (irrespective of the prior ${\pi }_2$ over ${\sigma }^2$); this is equivalent to minimising the utility:
  $U_{\text{Gauss}}\left(X\right)=-{\int }_Y{\int }_{\Theta }(\beta -{\beta }_{\text{MEP}}{)}^{\prime }(\beta -{\beta }_{\text{MEP}}\left)p\right(y|\theta ,X\left){\pi }_1\right(\beta |{\sigma }^2)d\beta dy.$(3.8)
  Bayesian A-optimality clearly carries over easily to other settings (e.g. weighted least squares regression, generalised linear models); the negative squared error loss utility function $u_{\text{NSEL}}$ is well defined and the expected utility $U\left(X\right)$ can be approximated using either deterministic numerical integration techniques or Monte Carlo methods.

• Pseudo-Bayesian-A optimality Following (3.2), a Bayesian A-optimal design for estimating a vector $\gamma \left(\theta \right)$ whose components are functions of the model parameters $\theta$, maximises:

  $U\left(X\right)=-{\int }_Y{\int }_{\Theta }(\gamma -E[\gamma |y,X\left]{)}^{\prime }\right(\gamma -E[\gamma |y,X]\left)p\right(y|\theta ,X\left)\pi \right(\theta )d\theta dy,$
  where the expectation is with respect to the posterior. In most situations that are different from the Gaussian setting where $\gamma =\beta$, this problem presents serious computational difficulties and hence the utility $u_{\text{NSEL}}$ is replaced by a utility that does not depend on $y$:
  $u_{\text{NSEL,A}}(\gamma ,y,X):=-\text{tr}\left(I^{-1}(X,\gamma )\right).$
  A design that is optimal with this utility function is a pseudo-Bayesian-A optimal design. The $A_B$-optimal design minimises:
  $A_B={\int }_{\Theta }\text{trace}\left(I^{-1}\right(X,\gamma \left)\pi \right(\theta )d\theta .$(3.9)

• C-optimality In many situations, it is not the parameters in and of themselves that are of interest, but rather functions of the parameters (these functions may be linear or non-linear). As we pointed out, in WTP experiments, with a vector of parameters $\beta$, ${\beta }_0$ is often the coefficient of price, while ${\beta }_1,\dots ,{\beta }_p$ represents non-price attributes. We may be interested in $\gamma =({\gamma }_1,\dots ,{\gamma }_p{)}^{\prime }$, where ${\gamma }_i=-\frac{{\beta }_i}{{\beta }_0}$, which measures the respondent’s WTP per unit cost for non-price attribute $i$.

  A C-optimal design is a design that is optimal for functions of the parameters, namely a design that is optimal for $\gamma$ rather than for $\beta$. This could be D- or A-optimality for $\gamma$, depending on preference.

• E-optimality In the classical setting, for the Gaussian model, E-optimality is defined as choosing the design which minimises the largest eigenvalue of $(X^{\prime }X{)}^{-1}$, which corresponds to minimising the maximum eigenvalue of $\text{Cov}\left(\hat{\beta }\right)$. Equivalently, we take the design which minimises

  $\underset{e:\sum _i{e}_i^2=1}{max}\text{Var}\left(\sum _i{e}_i{\hat{\beta }}_i\right)=e^{\prime }\text{Cov}\left(\hat{\beta }\right)e={\sigma }^2{e}^{\prime }(X^{\prime }X{)}^{-1}e.$(3.10)
  The maximising $e$ in (3.10) is the eigenvector of $\text{Cov}\left(\hat{\beta }\right)$ corresponding to the largest eigenvalue.

  More generally, to find an E-optimal design for a vector $\gamma$ (where each component of the vector is a function of the parameters), find the design $X$ which maximises the utility function:

  $u(\theta ,X)=\underset{e:\parallel e\parallel =1}{max}\left(-e^{\prime }\left(E_{\theta }\left({\nabla }_{\gamma }{\nabla }_{\gamma }log\pi (\theta |X,y)\right)\right)e\right),$(3.11)
  where $\parallel e{\parallel }^2=\sum _i{e}_i^2$. This $e$ is an eigenvector corresponding to the largest eigenvalue. In the classical setting, a reference parameter value $\theta$ is used (i.e. a particular parameter value chosen by the user). For Bayesian E-optimality, a prior $\pi$ over $\theta$ is incorporated in the usual way.

• S-optimality This criterion considers the mutual column orthogonality of $X$ and the determinant of the information matrix.

3.2.1 Example: Simple Linear Regression and Alphabet Optimality

In this Element, the emphasis is on optimal designs for choice experiments, which typically present the subject with two or three choices (alternative 1, alternative 2, often there is also the possibility to reject both the alternatives) which fall within the framework of generalised linear models. The Fisher information takes the form $X^{\prime }W\left(\beta \right)X$, where $W\left(\beta \right)$ is a diagonal matrix of weights. To illustrate the strengths and weaknesses of these optimality criteria, the computations are simplified enormously if $W\left(\beta \right)=I$ (i.e. the identity matrix) and where $X$ is simply an $n\times 2$ matrix ($n$ observations with two parameters). The general principles can therefore all be observed by considering the simple linear regression model fitting only two parameters (intercept and slope), where the optimality criteria are based on the $2\times 2$ matrix $X^{\prime }X$. This shows forth the general principle that, while optimal designs give optimality provided the model is correct, they give absolutely no possibility for model checking. For illustration, we therefore limit consideration to the simple linear regression model $Y=X\beta +ϵ$, where

$X=\left(\begin{array}{cc}1& x_1\\ ⋮& ⋮\\ 1& x_n\end{array}\right),\beta =\left(\begin{array}{c}{\beta }_0\\ {\beta }_1\end{array}\right)ϵ\sim N(0,{\sigma }^2{I}_n),$

and show properties of the optimal design for the various criteria. This is the model for a single covariate; a response $Y\left(x\right)$ when the covariate takes value $x$ is modelled as $Y\left(x\right)={\beta }_0+{\beta }_{1}x+ϵ$ where the $ϵ$’s are independent for different runs and $ϵ\sim N(0,{\sigma }^2)$.

The basic message is that if the model is known, then designs that are optimal according to alphabet-optimality criteria will produce the sharpest estimators for the relevant parameters (according to the selected criteria), but the designs may be rather poor if the model itself is uncertain and may not lend themselves to diagnostics for checking the model.

For simple linear regression, $X^{\prime }X=n\left(\begin{array}{cc}1& \bar{x}\\ \bar{x}& \overline{x^2}\end{array}\right)$ where $\bar{x}=\frac{1}{n}\sum _{j=1}^n{x}_j$ and $\overline{x^2}=\frac{1}{n}\sum _{j=1}^n{x}_j^2$. Using $\text{Var}\left(x\right)=\overline{x^2}-{\bar{x}}^2$, we have: $(X^{\prime }X{)}^{-1}=\frac{1}{n\text{Var}\left(x\right)}\left(\begin{array}{cc}\overline{x^2}& -\bar{x}\\ -\bar{x}& 1\end{array}\right)$ so that, for the classical setting,

$\left(\begin{array}{c}{\hat{\beta }}_0\\ {\hat{\beta }}_1\end{array}\right)\sim N\left(\left(\begin{array}{c}{\beta }_0\\ {\beta }_1\end{array}\right),\frac{{\sigma }^2}{n\text{Var}\left(x\right)}\left(\begin{array}{cc}{\bar{x}}^2+\text{Var}\left(x\right)& -\bar{x}\\ -\bar{x}& 1\end{array}\right)\right).$

For the estimates of ${\beta }_0$, ${\beta }_1$ and their covariance, this gives:

$\text{Var}\left({\hat{\beta }}_0\right)={\sigma }^2\left(\frac{\overline{x^2}}{n\text{Var}\left(x\right)}+\frac{1}{n}\right),\text{Var}\left({\hat{\beta }}_1\right)=\frac{{\sigma }^2}{n\text{Var}\left(x\right)},\text{Cov}({\hat{\beta }}_0,{\hat{\beta }}_1)=-\frac{{\sigma }^2\bar{x}}{n\text{Var}\left(x\right)}.$

We can see that $\text{Var}\left(x\right)$ is important here; if all the values $x_1,\dots ,x_n$ are approximately equal to $\bar{x}$ (so that $\text{Var}\left(x\right)$ is small), then the variances of the parameter estimators will be large.

For the Bayesian setting, we take a prior distribution $\pi \left(\beta \right)\sim N({\beta }_0,{\sigma }^2{\Omega }_0^{-1})$ for a fixed $2\times 2$ invertible matrix ${\Omega }_0$ (with a prior over ${\sigma }^2$) which updates to:

$\pi (\beta |y,X)\sim N\left((X^{\prime }X+{\Omega }_0{)}^{-1})({\Omega }_0{\beta }_0+X^{\prime }y),{\sigma }^2(X^{\prime }X+{\Omega }_0{)}^{-1}\right).$

Let ${\Omega }_0=\left(\begin{array}{cc}{\omega }_{00}& {\omega }_{01}\\ {\omega }_{01}& {\omega }_{11}\end{array}\right)$ so that

$X^{\prime }X+{\Omega }_0=\left(\begin{array}{cc}n+{\omega }_{00}& n\bar{x}+{\omega }_{01}\\ n\bar{x}+{\omega }_{01}& n\overline{x^2}+{\omega }_{11}\end{array}\right).$

Let $a=\frac{n\bar{x}+{\omega }_{01}}{n+{\omega }_{00}}$ and $b=\frac{n\overline{x^2}+{\omega }_{11}}{n+{\omega }_{00}}$ and $v=b-a^2$. Then

${\left(X^{\prime }X+{\Omega }_0\right)}^{-1}=\frac{1}{(n+{\omega }_{00})v}\left(\begin{array}{cc}{a}^2+v& -a\\ -a& 1\end{array}\right).$(3.12)

Note

In this set-up, the prior can be considered as information from an independent ‘virtual’ sample of size ${\omega }_{00}$, where the average of the covariate values is $\frac{{\omega }_{01}}{{\omega }_{00}}$ (corresponding to $\bar{x}$ for the virtual sample) and the variance of the covariate values is $\frac{{\omega }_{11}}{{\omega }_{00}}-{\left(\frac{{\omega }_{01}}{{\omega }_{00}}\right)}^2$ (corresponding to $\text{Var}\left(x\right)$ for the virtual sample). The quantity $a$ is therefore the overall average of all covariate values (virtual and actual sample), while $v$ is the overall variance of all covariate values (virtual and actual sample).

Let us consider the various optimality criteria for this example, both for the classical setting and for the Bayesian setting described earlier in this Element.

• D-optimality For the classical setting,

  $\text{det}\left(\text{Cov}\right(\hat{\beta }\left)\right)=\frac{{\sigma }^4}{n^2\text{Var}\left(x\right)},$
  while, in the Bayesian setting (taking $\hat{\beta }$ as a random vector whose distribution is the posterior over $\beta$) and using the notation defined in Equation (3.12),
  $\text{det}\left(\text{Cov}\right(\hat{\beta }\left)\right)=\frac{{\sigma }^4}{(n+{\omega }_{00}{)}^{2}v}.$
  Here, D-optimality gives us something quite intuitive; we should find a design which maximises $\text{Var}\left(x\right)$ in the classical setting and $v$ in the Bayesian.

An Obvious Problem

While D-optimality gives a design that is optimal provided the model is correct, it delivers a design that is useless for model checking. Indeed, suppose we have a covariate $x$ which takes possible values in $\{1,\dots ,9\}$ and we have an even number of observations $n=2m$. The optimal design here is clear: take $m$ observations with covariate value $x=1$ and $m$ observations with covariate value $9$. This is the design that maximises $\text{Var}\left(x\right)$ (and hence minimises the objective).

With such a design, where only two values of the covariate are considered (at the extreme ends, in order to maximise $\text{Var}\left(x\right)$), it is impossible to check whether the linear model $Y\left(x\right)={\beta }_0+{\beta }_{1}x+ϵ$ in fact holds; the D-optimal design only gives us information for values $x=1$ and $x=9$ and we are left without any information at all to apply the standard diagnostics for checking whether the straight line plus IID noise is actually a good model. If the true model were $Y={\beta }_0+{\beta }_{1}x+{\beta }_2{x}^2+ϵ$, we would have no way of checking this with the data from a D-optimal design for the model $Y={\beta }_0+{\beta }_{1}x+ϵ$.

If there is any doubt about whether a reduced model is suitable, the D-optimal design for the full model should be computed. In the experimental design setting, this means (for example) that if we want to test whether there are important interaction effects, they have to be included in the model when computing the D-optimal design.

• A-optimality The A-optimality criterion ignores possible covariance between the parameter estimates; in this case, an A-optimal design minimises the trace of the covariance. In the classical setting, this means minimising:

  $\frac{\text{Var}\left(x\right)+{\bar{x}}^2+1}{\text{Var}\left(x\right)}=1+\frac{{\bar{x}}^2+1}{\text{Var}\left(x\right)}.$
  Again, if the covariate values are chosen in an interval $[z_1,z_2]$, then the optimal design chooses a proportion $\alpha$ with covariate value $z_1$ and a proportion $1-\alpha$ with covariate value $z_2$, so that $\bar{x}=\alpha z_1+(1-\alpha )z_2$. The solution is obtained by finding the $\alpha$ that minimises:
  $\frac{1+(\alpha z_1+(1-\alpha )z_2{)}^2}{\alpha (1-\alpha )(z_1-z_2{)}^2}.$
  For model checking, an A-optimal design has the same problems as a D-optimal design; the design that minimises the objective does not permit diagnostics which check whether the model is correct.

• C-optimality We consider C-optimality in a straightforward linear setting, for linear functions of the parameters. We want to minimise $\text{Var}(c_0{\hat{\beta }}_0+c_1{\hat{\beta }}_1)$ for a given $(c_0,c_1)$. For example, suppose we are interested in precise estimates of the average value of future observations with covariate value $z$. Here $E\left[Y\right(z\left)\right]={\beta }_0+{\beta }_{1}z$, so that $(c_0,c_1)=(1,z)$. If the aim is to obtain sharp estimates on $E[c_0{\hat{\beta }}_0+c_1{\hat{\beta }}_1]$, then we choose $x_1,\dots ,x_n$ to minimise:

  $\begin{array}{cc}\text{Var}(c_0{\hat{\beta }}_0+c_1{\hat{\beta }}_1)& =\frac{{\sigma }^2}{n\text{Var}\left(x\right)}\left(c_0^2{\bar{x}}^2+c_0^2\text{Var}\left(x\right)+c_1^2-2c_0{c}_1\bar{x}\right)\\ & =\frac{{\sigma }^2}{n}\left(c_0^2+\frac{(c_0\bar{x}-c_1{)}^2}{\text{Var}\left(x\right)}\right).\end{array}$
  Any design satisfying $c_0\bar{x}=c_1$ (e.g. choose $x_1=\dots =x_n=\frac{c_1}{c_0}$) minimises the objective. This is not a big surprise; if we want to estimate $E\left[Y\right(z\left)\right]={\beta }_0+{\beta }_{1}z$, so that $c_0=1$ and $c_1=z$ (and we are not interested in any other covariate value), then the natural experiment would be $n$ repetitions, where $x_1=\dots =x_n=z$.

  Bayesian C-optimality, for a fixed $c=(c_0,c_1)$ requires the design $(x_1,\dots ,x_n)$ which minimises:

  $\frac{{\left(c_{0}a-c_1\right)}^2}{v},$
  where $a=\frac{n\bar{x}+{\omega }_{01}}{n+{\omega }_{00}}$ and $v$ are defined for Equation (3.12). Choosing any design such that $\bar{x}=\frac{\frac{c_1}{c_0}(n+{\omega }_{00})-{\omega }_{01}}{n}$ will satisfy C-optimality for a particular linear combination $c=(c_0,c_1)$.

• E-optimality Returning to Equations (3.10) and (3.12) (considering the Bayesian setting), an E-optimal design is a design which minimises:

  $\frac{1}{(n+{\omega }_{00})v}\underset{(e_0,e_1):e_0^2+e_1^2=1}{max}(e_0,e_1)\left(\begin{array}{cc}{a}^2+v& -a\\ -a& 1\end{array}\right)\left(\begin{array}{c}{e}_0\\ e_1\end{array}\right).$
  The maximising $(e_0,e_1)$ is the eigenvector corresponding to the largest eigenvalue which (of course) depends on $a$ and $v$. While D-optimality considers the determinant, which is the product of eigenvalues, and A-optimality considers the trace, which is the sum of eigenvalues, E-optimality reduces the problem by selecting a single eigenvalue, which is the largest eigenvalue.

  The computation is omitted; the expression to be minimised then depends only on $a$ and $v$, decreasing in $v$, which turns out (unsurprisingly) to have the same properties as $D$ and $A$ optimisation; if the values are restricted to an interval $x_i\in [l_{-},l_{+}]$, the E-optimal design will choose extreme values to maximise $\text{Var}\left(x\right)$ (Classical) or $v$ (Bayesian, from (3.12)); a proportion $\alpha$ will satisfy $x_i=l_{-}$; and a proportion $1-\alpha$ will satisfy $x_i=l_{+}$. As with other criteria, the E-optimal design therefore restricts itself to two values and does not give any possibility for model checking.

• S-optimality Column orthogonality means that we would like to choose our covariates $x_1,\dots ,x_n$ such that $\bar{x}=0$ (i.e. $({\text{1}}_n,x)=0$). For such designs, the whole business simplifies and:

  $(X^{\prime }X{)}^{-1}=\frac{1}{n}\left(\begin{array}{cc}1& 0\\ 0& \frac{1}{\overline{x^2}}\end{array}\right).$
  Column orthogonality gives: ${\hat{\beta }}_0\perp {\hat{\beta }}_1$ (i.e. the estimators are independent). Also, with such column orthogonality, ${\hat{\beta }}_0=\bar{Y}$ (simply the sample average) and the only design consideration is, within the constraint $\bar{x}=0$, to maximise $\overline{x^2}$ to get the optimal estimator of ${\beta }_1$.

  Designs which have column orthogonality (orthogonal contrast designs such as $2^k$ experiments) are often easier to deal with.

We now illustrate the criteria which are based on the predictive distribution with this simple two-parameter example, noting that all the basic principles for a more general setting are already present in the two-parameter example.

• G-optimality Consider a new observation where the covariate is $z$; consider ${\sigma }^2$ fixed and known. $Y$ is the $n$-vector of observations and $X$ is the design matrix. Hence, we use $\pi (\beta |X,Y)$ to denote the distribution of the estimator of $\beta$. This is the distribution of $\hat{\beta }$, the maximum likelihood estimator, in the classical case and the posterior distribution in the Bayesian setting. The predictive distribution of $y\left(z\right)$ (new observation $y$ with covariate value $z$, using $Y$ to denote the $n$-vector of observations on which the parameter estimates are based) is:

  $p\left(y|z\right)=\frac{\int \pi (\beta |Y,X)p(y|\beta ,z)d\beta }{\int \int \pi (\beta |Y,X)p(y|\beta ,z)d\beta dy}.$
  In the classical setting, $\pi (\beta |y,X)\sim N(\hat{\beta },{\sigma }^2(X^{\prime }X{)}^{-1})$. For covariate value $z$, a new observation $y$ has distribution $p(y|\beta ,z)\sim N({\beta }_0+{\beta }_{1}z,{\sigma }^2)$. The density of the predictive distribution is therefore:
  $p(y|X,Y,z)\sim N\left({\hat{\beta }}_0+{\hat{\beta }}_{1}z,{\sigma }^2\left(1+\frac{1}{n}+\frac{(\bar{x}-z{)}^2}{n\text{Var}\left(x\right)}\right)\right).$
  In the Bayesian setting, the predictive distribution is:
  $p(y|X,Y,z)\sim N\left({\beta }_{0,\text{MEP}}+{\beta }_{1,\text{MEP}}z,{\sigma }^2\left(1+\frac{1}{n+k}+\frac{(A-z{)}^2}{(n+k)B}\right)\right),$
  where $\left(\begin{array}{c}{\beta }_{0,\text{MEP}}\\ {\beta }_{1,\text{MEP}}\end{array}\right)$ is the mean of the posterior $\pi (\beta |X,Y,z)$. The G-optimality criterion is therefore the same as C-optimality, where $c=(1,z)$, and is therefore useless for producing a design that allows diagnostics to check whether the model is reasonable.

• I- and V-optimality These are the same as G-optimality, except that we have a reduced set of designs over which to make the optimisation.