The phrases modeling and simulation and computational simulation are becoming prevalent in a wide variety of technical, economic, governmental, and business activities (Schrage, 1999). Indeed, the phrases are becoming so common that one is even beginning to see them in the mass media. What do they mean? These phrases can have a wide variety of meanings depending on the field and the context. Here, we are concerned with the fields of the physical sciences and engineering. By examining the fundamentals of modeling and simulation (M&S) and scientific computing, our goal is to see the similarities in model formulation and computational issues across a wide range of physical systems. Our approach to scientific computing emphasizes the similarities that exist in mathematical form and structure of models in many technical disciplines. Then a framework, an overarching structure, is constructed for either attacking more detailed features of the system or for attacking more complex systems. Commonly, more complex systems involve coupling different types of physical phenomena, incorporation of additional elements from the system or the surroundings, and effects of human intervention in the system. Similarities in model formulation issues exist because many of the model properties are not determined by their physical nature, but by their mathematical structure, the interaction of the system with the surroundings, and the similarity in the nature of the system responses.

Many of the difficult issues that must be dealt with in verification, validation, and uncertainty quantification (VV&UQ) can be traced back to ambiguities and inconsistencies in the model formulation, the mapping of continuum mathematics models to discrete mathematics models, and vague or improper characterizations of uncertainties. This chapter deals with many of those issues by examining the fundamentals of M&S.

As has been discussed in a number of chapters, particularly Chapter 10, Model validation fundamentals, and Chapter 11, Design and execution of validation experiments, model accuracy assessment is the core issue of model validation. Our intent in model accuracy assessment is to critically and quantitatively determine the ability of a mathematical model and its embodiment in a computer code to simulate a well-characterized physical process. We, of course, are only interested in well-characterized physical processes that are useful for model validation. How critical and quantitative the model accuracy assessment is will depend on (a) how extensive the experimental data set is in exploring the important model input quantities that affect the system response quantities (SRQs) of interest; (b) how well characterized the important model input quantities are, based on measurements in the experiments; (c) how well characterized the experimental measurements and the model predictions of the SRQs of interest are; (d) whether the experimental measurements of the SRQs were available to the computational analyst before the model accuracy assessment was conducted; and (e) if the SRQs were available to the computational analysts, whether they were used for model updating or model calibration. This chapter will explore these difficult issues both conceptually and quantitatively.

We begin the chapter by discussing the fundamental elements of model accuracy assessment. As part of this discussion, we review traditional and recent methods for comparing model results and experimental measurements, and we explore the relationship between model accuracy assessment, model calibration, and model prediction. Beginning with the engineering society definitions of terms given in Chapter 2, Fundamental concepts and terminology, the perspective of this book is to segregate, as well as possible, each of these activities. There is, however, an alternative perspective in the published literature that believes all of these activities should be combined. We briefly review this alternative perspective and the associated approaches, and contrast these with approaches that segregate these activities.

This chapter will synthesize the key results from the previous chapters and incorporate them into modern predictive capability in scientific computing. This chapter, in contrast to all other chapters, does not stress the theme of assessment. Here we discuss the fundamental steps in conducting a nondeterministic analysis of a system of interest. With this discussion we show how verification and validation (V&V) can directly contribute to predictive capability.

The previously covered material and the new material are organized into six procedural steps to make a prediction:

1. identify all relevant sources of uncertainty,

2. characterize each source of uncertainty,

3. estimate numerical solution error in the system response quantities of interest,

4. estimate uncertainty in the system response quantities of interest,

5. conduct model updating,

6. conduct sensitivity analysis.

All of these steps, except step 3, are widely practiced in nondeterministic simulations and risk analysis. Step 3 is not commonly addressed for three reasons. First, in many simulations the numerical solution error is assumed to be small compared to other contributors to uncertainty. Sometimes this assumption is quantitatively justified, and sometimes it is simply posited with little or no evidence. Second, in some computationally intensive simulations, it is understood that the numerical solution error is important, and possibly even dominant, but it is argued that various modeling parameters can be adjusted to compensate for the numerical error. If the application of interest is sufficiently similar to the conditions for which experimental data are available, then it is claimed that the adjustable parameters can be used to match the existing data and thereby make reasonable predictions. Third, even if the numerical error is estimated, and it is not small relative to other uncertainties, there are no generally accepted procedures for including its effect on the system response quantities (SRQs) of interest.

In scientific computing, the goal of code verification is to ensure that the code is a faithful representation of the underlying mathematical model. This mathematical model generally takes the form of partial differential or integral equations along with associated initial condition, boundary conditions, and auxiliary relationships. Code verification thus addresses both the correctness of the chosen numerical algorithm and the correctness of the instantiation of that algorithm into written source code, i.e., ensuring there are no coding mistakes or “bugs.”

A computer program, referred to here simply as a code, is a collection of instructions for a computer written in a programming language. As discussed in Chapter 4, in the software engineering community code verification is called software verification and is comprised of software tests which ensure that the software meets the stated requirements. When conducting system-level testing of non-scientific software, in many cases it is possible to exactly determine the correct code output for a set of given code inputs. However, in scientific computing, the code output depends on the numerical algorithm, the spatial mesh, the time step, the iterative tolerance, and the number of digits of precision used in the computations. Due to these factors, it is not possible to know the correct code output (i.e., numerical solution) a priori. The developer of scientific computing software is thus faced with the difficult challenge of determining appropriate system-level software tests.

Because of the well-developed methodologies and techniques developed in metrology, experimental measurements are usually viewed as the best way to estimate a true value, if one exists. This level of trust and credibility in experimental measurements has been built over at least four millennia of learning, both from mistakes and successes. This is not meant to imply that experimental measurements always yield an accurate estimate of the true value. Experimental measurements can be inaccurate, or downright wrong, for many reasons. What is meant by the trustworthiness of an experimental measurement is that techniques for investigating its limitations, weaknesses, and uncertainties are generally well understood. When new experimental diagnostic techniques are developed they must be carefully investigated and understood. Where possible, comparisons need to be made with measurements from existing, better understood, techniques so that measurement uncertainty can be better quantified. As science and technology progresses, new measurement techniques can increase the confidence in measurement accuracy and also begin to measure physical quantities that were previously immeasurable.

An enlightening way of thinking about the trustworthiness of experimental measurements is to think of an experimental measurement as “asking a question of nature” (Hornung and Perry, 1998). When a measurement result is obtained, the result can be thought of as nature's answer to a question we have asked. We tend to believe that the answer obtained is the answer to the question we think we asked. However, this is not actually the case because there are always assumptions involved on our part. For example, when we measure the fluid velocity in a flow field, we believe we asked the question: given the flow field of interest, what is the velocity of the fluid at a certain point? Our intent is to ask nature the question that excludes or minimizes the effect of random or systematic errors in the measurement. If, however, there is significant random measurement error or if there is an unknown systematic error either in the measurement itself or in the data reduction procedure, then the question asked of nature is different from what we thought we asked. That is, nature answered the question that includes the random and systematic errors, whether large or small.

This final section of the book primarily deals with the topic of how managers – both line managers and project managers – plan, implement, develop, and sustain verification, validation, and uncertainty quantification (VV&UQ) capabilities in their organizations. Some readers may feel this topic is inappropriate in a book on scientific computing; however, our experience and the experience of others has convinced us that while technical issues and computing resources are important, they are not the limiting factor in improving the credibility and usefulness of scientific computing used in a decision-making environment. We point out that although computing speed has continued to increase by a factor of ten every four years for multiple decades, we do not believe there has been a comparable impact of the information produced in modeling and simulation (M&S). We believe that nontechnical issues have significantly constrained improvements in the credibility of the information produced in M&S. Examples of these issues are (a) poor allocation of resources relative to the simulation needs of a project, (b) inadequate and ambiguous characterization and understanding of uncertainties in simulations, and (c) the difficulty of management and staff to assess how the time and resources invested in VV&UQ produce a net benefit for the credibility of the simulation results produced.

Recommended programming practices

The following is a list of recommended programming practices designed to increase the reliability of scientific computing software, along with a brief description of each practice.

Use strongly-typed programming languages

Although there is some ambiguity in the definition, here we refer to a strongly-typed programming language as one which (1) requires that a variable or object maintain the same type (e.g., integer, floating point number, character) during program execution and (2) has strict rules as to which types can be used during operations (i.e., implicit type conversions are not allowed). Common examples of the latter are the use of integer division on floating point numbers and the use real functions (e.g., cos, log) on integers. BASIC and C are considered weakly-typed languages, while C++, Java, Fortran, Pascal, and Python are considered strongly-typed. For type information on other programming languages, see en.wikipedia.org/wiki/Template:Type_system_cross_reference_list. A type-safe program can be written in a weakly-typed language by using explicit type conversions (to convert integers to real numbers, real numbers to integers, etc.). In other words, explicit type conversions should be used, even when not required by the programming language.

In Chapter 2, Fundamental concepts and terminology, a summary discussion was given concerning an integrated view of verification, validation, and prediction. This chapter will discuss in detail Element 2, Planning and prioritization of activities, as shown in Figure 14.1. The topic of this chapter could be summarized as: given the wide range of activities dealing with modeling and simulation, how does one allocate resources to best achieve the goals of the project? Here, we are interested in dealing with the perspective of the management responsibilities for a large-scale, project-oriented activity as opposed to a research effort or general capability development of a commercial software package. Our discussion applies to projects within both industry and government. We emphasize engineering system projects, but the discussion also applies to the analysis of natural systems, e.g., underground storage of radioactive wastes, global climate change, and transport of contaminants or chemical agents due to atmospheric winds. The system of interest could be a new or proposed system in the design phase, an existing system that is being considered for modification or upgrade, or analysis of a system as it presently exists.

Methodology for planning and prioritization

For large-scale projects, planning and prioritization (P&P) are major activities that require a significant investment in time, money, and specialized personnel talent. Although we are primarily concerned with P&P in V&V and prediction, we comment here on the importance of P&P in large-scale projects for the development of an M&S capability. Depending on the type of organization developing the capability, it can be challenging for management to assemble the personnel with the needed talent and interest in P&P. For example, if a large-scale M&S project arrives in the midst of ongoing computational development activities, it can be difficult for management and staff to change pre-existing viewpoints, traditions and ingrained habits that are no longer appropriate, and possibly even very detrimental, to the goals of the new project. Needed changes can be particularly onerous to affect in a research or government organization.

Chapter 10, Model validation fundamentals, discussed the philosophy of validation experiments and how they differ from traditional experiments as well as calibration experiments. A validation experiment is conducted for the primary purpose of determining the predictive accuracy of a mathematical model. In other words, a validation experiment is designed, executed, and analyzed for the purpose of quantitatively determining the ability of a mathematical model and its embodiment in a computer code to simulate a well-characterized physical process. In this chapter we describe six primary guidelines for the design and execution of validation experiments. Our discussion will also deal with how these guidelines can be carried out and why they are sometimes difficult to execute in practice.

We will then discuss a high-quality validation experiment that the authors, as well as several others, designed and executed. The experiment is referred to as the Joint Computational/Experimental Aerodynamics Program (JCEAP). From the beginning of the project, it was designed to synergistically couple computational fluid dynamics development and an experimental research in a hypersonic wind tunnel. The program was initiated in 1990 at Sandia National Laboratories and came to a successful close in 1997. The program helped develop the six fundamental guidelines for the design and execution of high-quality validation experiments. We use JCEAP to exemplify these six guidelines and make recommendations for how they can be applied to validation experiments in general.

Software engineering encompasses the tools and methods for defining requirements for, designing, programming, testing, and managing software. It consists of monitoring and controlling both the software processes and the software products to ensure reliability. Software engineering was developed primarily from within the computer science community, and its use is essential for large software development projects and for high-assurance software systems such as those for aircraft control systems, nuclear power plants, and medical devices (e.g., pacemakers).

The reader may wonder at this point why a book on verification and validation in scientific computing includes a chapter on software engineering. The reason is that software engineering is critical for the efficient and reliable development of scientific computing software. Failure to perform good software engineering throughout the life cycle of a scientific computing code can result in much more additional code verification testing and debugging. Furthermore, it is extremely difficult to estimate the effect of unknown software defects on a scientific computing prediction (e.g., see Knupp et al., 2007). Since this effect is so difficult to quantify, it is prudent to minimize the introduction of software defects through good software engineering practices.

Chapter 2, Fundamental Concepts and Terminology and Chapter 3, Modeling and Computational Simulation form the foundation of the book. These chapters are recommended reading for individuals interested in any aspect of verification and validation (V&V) of mathematical models and scientific computing simulations. In Chapter 2, all of the key terms are defined and discussed. The chapter is much more than a glossary because it describes the development of the terminology and the underlying philosophical principles of each concept. The reader may be surprised that this chapter is devoted to fundamental concepts and terminology; however, understanding the underlying concepts is critical because many of the terms (e.g., verification, validation, predictive capability, calibration, uncertainty, and error) have a common language meaning that is imprecise and some terms are even contradictory from one technical field to another. One of the exciting aspects of the new field of V&V is that all of the principles developed must be applicable to any field of scientific computing, and even beyond. This is also challenging, and at times frustrating, because the terminology from various technical fields can be at odds with the terminology that is developing in the field of V&V. The discussion presents clear arguments why the concepts and terminology are logical and useable in real applications of scientific computing. Chapter 2 closes with an in-depth discussion of a framework of how all of the aspects of V&V and predictive capability are related and sequentially accomplished.

This chapter discusses some perspectives on the responsibilities for the development, conduct, delivery, and management of verification, validation, and uncertainty quantification (VV&UQ). The topics considered here deal with both technical and nontechnical issues, but all are focused on the strategic goals of improved credibility and proper use of simulations. Our experience, and the experience of others, has convinced us that while technical issues and computing resources are important, they are not the limiting factor in improving the credibility and usefulness of scientific computing used in a decision-making environment. We believe that nontechnical issues have significantly constrained the improvements in credibility that VV&UQ can provide.

Examples of the issues discussed in this chapter are (a) suggestions for needed technical developments in VV&UQ, (b) responsibilities for the various technical activities that encompass VV&UQ, (c) recommendations for management responsibilities and leadership in deploying VV&UQ practices, (d) development of V&V databases for general use, and (e) development of industrial and engineering standards for V&V.

This section of the book presents an in-depth discussion of the topic of model validation and prediction. As discussed in Part I, this book uses the restricted meaning of model validation, i.e., assessment of model accuracy as determined by comparison of model outputs with experimental measurements. Stated differently, if you are not comparing model predictions of system response quantities (SRQs) with experimental measurements of the SRQs for the purpose of assessing the accuracy of the model, you are not conducting model validation. Prediction, as discussed in Part I, deals with the use of the model and all information that is available concerning the system of interest, as well has how well the model has performed in model validation activities, to predict the response of the system of interest for which no experimental data is presently available. That is, we use the model to foretell the response of the system of interest given our knowledge of the system, and how well the model has compared with available experimental measurements, including our estimates of all of the uncertainties involved in every element of the simulation.

The previous chapter focused on the estimation of numerical errors and uncertainties due to the discretization. In addition to estimating the discretization error, we also desire methods for reducing it when either it is found to be too large or when the solutions are not yet in the asymptotic range and therefore the error estimates are not reliable. Applying systematic mesh refinement, although required for assessing the reliability of all discretization error estimation approaches, is not the most efficient method for reducing the discretization error. Since systematic refinement, by definition, refines by the same factor over the entire domain, it generally results in meshes with highly-refined cells or elements in regions where they are not needed. Recall that for 3-D scientific computing applications, each time the mesh is refined using grid halving (a refinement factor of two), the number of cells/elements increases by a factor of eight. Thus systematic refinement for reducing discretization error can be prohibitively expensive.

Targeted, local solution adaptation is a much better strategy for reducing the discretization error. After a discussion of factors affecting the discretization error, this chapter then addresses the two main aspects of solution adaptation:

1. methods for determining which regions should be adapted and

2. methods for accomplishing the adaption.

Introduction

This chapter presents a detailed introduction to various finite precision representations of rational numbers and associated arithmetic. It is the result of more than a decade of collaborative research by the authors, and as such has not so far been given a cohesive presentation, but has only been available in a sequence of conference presentations and journal papers. These representations are well founded in classical mathematics (number theory), and many problems are naturally stated in and computed in the domain of rationals. Such problems can be solved exactly in systems based on unbounded integer representations, and possibly also in residue representations as discussed in the previous chapter. However, in many cases an approximate solution based on a finite representation may be sufficient.

The first class of representations is based on the traditional way of using pairs of integers of bounded size. Two natural bounds are proposed: the first simply applying a common bound on the values of numerator and denominator, the other a bound on the product of these. The latter representation will allow a larger range of representable rational values, but for practical reasons it will be approximated by bounding the total number of digits (bits) employed for representing the numerator and denominator using radix polynomials.

It is essential in all such finite representation systems to have an associated rounding rule which allows the mapping of any value from the underlying infinite mathematical domain into the chosen finite set of representable values.

Introduction

Just as the radix representation is the ubiquitous representation of numbers, so radix integer addition is the most fundamental operation for all computer arithmetic; it is used in the implementation of all the other standard arithmetic operations: subtraction, multiplication, division, and square root. For a discussion of the implementation of the standard arithmetic operators on simple radix-represented operands, it is sufficient to consider integer operands. The modifications necessary to extend the algorithms and hardware implementations to other types of fixed-point operands are very trivial. Also the problem of subtraction is trivially reduced to addition, so we shall only briefly cover subtraction in the following.

It is, however, crucial to the discussions in this chapter that we restrict ourselves to a finite set of integer values, as typically defined by the limitations of computer words or registers used in hardware implementations. Radix integer addition on such restricted operand domains then forms the most basic “building block” upon which other radix integer arithmetic operations can be built. Eventually we will also build other finite precision arithmetic systems on these radix integer operations, e.g., floating-point arithmetic and more unusual or exotic types of arithmetic based on other number representations, like rational representations. Even the types of “exact” or unlimited precision arithmetic found in symbolic and algebraic systems are based on these basic “building block” integer arithmetic operations.

Introduction

Conversion of a number from one radix representation into another plays an important role in computing, the most obvious example being the conversion between the binary representation and the everyday decimal representation that occurs in most I/O operations. But, as we shall see later, many arithmetic algorithms depend heavily on the ability to convert between radix systems, because some algorithms may be faster if performed in higher radices, and in particular if redundant digit sets are exploited.

There is amazingly little published in the open literature on these subjects in their own right; most of what can be found is usually either very trivial, or embedded in the description of some particular application.

There are, however, many such results, e.g., in multioperand addition or in multiplication the accumulation of many summands is often performed in a redundant representation, and only at the end converted into a non-redundant representation. This final digit set conversion is traditionally described as an addition process, but one may also consider ordinary addition a special case of digit set conversion.

The characteristic idea in many algorithms for multiplication often lies in converting one of the factors into a redundant representation in a higher radix, but may not really be described as such, rather it is seen as some kind of trick. Similarly, in digit serial division algorithms the quotient may be delivered in a redundant representation, but has to be converted before it is delivered as the result.