We first discuss a phenomenon called data mining. This can involve multiple tests on which variables or correlations are relevant. If used improperly, data mining may associate with scientific misconduct. Next, we discuss one way to arrive at a single final model, involving stepwise methods. We see that various stepwise methods lead to different final models. Next, we see that various configurations in test situations, here illustrated for testing for cointegration, lead to different outcomes. It may be possible to see which configurations make most sense and can be used for empirical analysis. However, we suggest that it is better to keep various models and somehow combine inferences. This is illustrated by an analysis of the losses in airline revenues in the United States owing to 9/11. We see that out of four different models, three estimate a similar loss, while the fourth model suggests only 10 percent of that figure. We argue that it is better to maintain various models, that is, models that stand various diagnostic tests, for inference and for forecasting, and to combine what can be learned from them.

Chapter 17 covers TWO-WAY INTERACTIONS IN MULTIPLE REGRESSION and includes the following specific topics, among others: Two-Way Interaction, First-Order Effects, Main Effects, Interaction Effects, Model Selection, AIC, BIC, and Probing Interactions.

Chapter 17 covers two-way interactions in multiple regression and includes the following specific topics, among others: two-way interaction, first-order effects, main effects, interaction effects, model selection, AIC, BIC, and probing interactions.

Modelling a neural system involves the selection of the mathematical form of the model’s components, such as neurons, synapses and ion channels, plus assigning values to the model’s parameters. This may involve matching to the known biology, fitting a suitable function to data or computational simplicity. Only a few parameter values may be available through existing experimental measurements or computational models. It will then be necessary to estimate parameters from experimental data or through optimisation of model output. Here we outline the many mathematical techniques available. We discuss how to specify suitable criteria against which a model can be optimised. For many models, ranges of parameter values may provide equally good outcomes against performance criteria. Exploring the parameter space can lead to valuable insights into how particular model components contribute to particular patterns of neuronal activity. It is important to establish the sensitivity of the model to particular parameter values.

We can easily find ourselves with lots of predictors. This situation has been common in ecology and environmental science but has spread to other biological disciplines as genomics, proteomics, metabolomics, etc., become widespread. Models can become very complex, and with many predictors, collinearity is more likely. Fitting the models is tricky, particularly if we’re looking for the “best” model, and the way we approach the task depends on how we’ll use the model results. This chapter describes different model selection approaches for multiple regression models and discusses ways of measuring the importance of specific predictors. It covers stepwise procedures, all subsets, information criteria, model averaging and validation, and introduces regression trees, including boosted trees.

In this chapter we introduce Bayesian inference and use it to extend the frequentist models of the previous chapters. To do this, we describe the concept of model priors, informative priors, uninformative priors, and conjugate prior-likelihood pairs . We then discuss Bayesian updating rules for using priors and likelihoods to obtain posteriors. Building upon priors and posteriors, we then describe more advanced concepts including predictive distributions, Bayes factors, expectation maximization to obtain maximum posterior estimators, and model selection. Finally, we present hierarchical Bayesian models, Markov blankets, and graphical representations. We conclude with a case study on change point detection.

Progress in the computational cognitive sciences depends critically on model evaluation. This chapter provides an accessible description of key considerations and methods important in model evaluation, with special emphasis on evaluation in the forms of validation, comparison, and selection. Major sub-topics include qualitative and quantitative validation, parameter estimation, cross-validation, goodness of fit, and model mimicry. The chapter includes definitions of an assortment of key concepts, relevant equations, and descriptions of best practices and important considerations in the use of these model evaluation methods. The chapter concludes with important high-level considerations regarding emerging directions and opportunities for continuing improvement in model evaluation.

This chapter discusses the problem of selecting predictors in a linear regression model, which is a special case of model selection. One might think that the best model is the one with the most predictors. However, each predictor is associated with a parameter that must be estimated, and errors in the estimation add uncertainty to the final prediction. Thus, when deciding whether to include certain predictors or not, the associated gain in prediction skill should exceed the loss due to estimation error. Model selection is not easily addressed using a hypothesis testing framework because multiple testing is involved. Instead, the standard approach is to define a criterion for preferring one model over another. One criterion is to select the model that gives the best predictions of independent data. By independent data, we mean data that is generated independently of the sample that was used to inform the model building process. Criteria for identifying the model that gives the best predictions in independent data include Mallows’ Cp, Akaike’s Information Criterion, Bayesian Information Criterion, and cross-validated error.

Multivariate linear regression is a method for modeling linear relations between two random vectors, say X and Y. Common reasons for using multivariate regression include (1) to predicting Y given X, (2) to testing hypotheses about the relation between X and Y, and (3) to projecting Y onto prescribed time series or spatial patterns. Special cases of multivariate regression models include Linear Inverse Models (LIMs) and Vector Autoregressive Models. Multivariate regression also is fundamental to other statistical techniques, including canonical correlation analysis, discriminant analysis, and predictable component analysis. This chapter introduces multivariate linear regression and discusses estimation, measures of association, hypothesis testing, and model selection. In climate studies, model selection often involves selecting Y as well as X. For instance, Y may be a set of principal components that need to be chosen, which is not a standard selection problem. This chapter introduces a criterion for selecting X and Y simultaneously called Mutual Information Criterion (MIC).

With rapid development in hardware storage, precision instrument manufacturing, and economic globalization etc., data in various forms have become ubiquitous in human life. This enormous amount of data can be a double-edged sword. While it provides the possibility of modeling the world with a higher fidelity and greater flexibility, improper modeling choices can lead to false discoveries, misleading conclusions, and poor predictions. Typical data-mining, machine-learning, and statistical-inference procedures learn from and make predictions on data by fitting parametric or non-parametric models. However, there exists no model that is universally suitable for all datasets and goals. Therefore, a crucial step in data analysis is to consider a set of postulated candidate models and learning methods (the model class) and select the most appropriate one. We provide integrated discussions on the fundamental limits of inference and prediction based on model-selection principles from modern data analysis. In particular, we introduce two recent advances of model-selection approaches, one concerning a new information criterion and the other concerning modeling procedure selection.

This chapter focuses on model evaluation and selection in Hierarchical Modelling of Species Communities (HMSC). It starts by noting that even if there are automated procedures for model selection, the most important step is actually done by the ecologist when deciding what kind of models will be fitted. The chapter then discusses different ways of measuring model fit based on contrasting the model predictions with the observed data, as well as the use of information criteria as a method for evaluating model fit. The chapter first discusses general methods that can be used to compare models that differ either in their predictors or in their structure, e.g. models with different sets of environmental covariates, models with and without spatial random effects, models with and without traits or phylogenetic information or models that differ in their prior distributions. The chapter then presents specific methods for variable selection, aimed at comparing models that are structurally identical but differ in the included environmental covariates: variable selection by the spike and slab prior approach, and reduced rank regression that aims at combining predictors to reduce their dimensionality.

This paper develops the idea that nosological reform is ultimately a matter of finding homogeneous groups of patients that are maximally distinct from each other. The focus lies on the statistical properties of patients, so that the problem of classification coincides with the problem of the reference class from the philosophy of science. It is argued that specific statistical methods – model selection and causal modeling – can assist in finding good classifications. An important advantage of these statistical methods is that they do not favor any particular explanatory level or vocabulary. Whether or not we should include some patient characteristic in our classification scheme is an empirical issue, to be settled entirely by its contribution to the performance of the scheme in predictions and intervention decisions. For this reason the paper adopts a so-called a-reductionist perspective: we do not need a principled discussion on reductionism.