Good software gets the immediate job done. But great software, written with a consistent and readable style, is predictable, robust, maintainable, supportable, and extensible. A few general principles provide the foundation for great software.

Avoid Causing Confusion

Avoid doing things that would be an unpleasant surprise to other software developers. The interfaces and the behavior exhibited by your software must be predictable and consistent. If they are not, then the documentation must clearly identify and justify any unusual instances of use or behavior.

To minimize the chances that anyone would encounter something surprising in your software, you should emphasize the following characteristics in its design, implementation, packaging, and documentation:

1. Simplicity Meet the expectations of your users with simple functions, classes, and methods.

2. Clarity Ensure that each variable, function, class, and method has a clear purpose.

3. Completeness Provide at least the minimum functionality that any reasonable user would expect to find and use.

4. Consistency Design similar entities with a similar look and behavior. Create and apply consistent standards whenever possible.

5. Robustness Provide predictable, documented behavior in response to errors and exceptions. Do not hide errors, and do not force users to detect errors.

Avoid Throw-Away Code

Apply these rules to any code you write, not just code destined for production. All too often, some piece of prototype or experimental code will make its way into a finished product. Even if your code never makes it into production, someone else may still have to read it.

General

There are large variations in the amount and character of documentation for software development projects. MATLAB-specific style issues are concentrated in the internal documentation intimately associated with the code files, particularly comments and reference pages. Documentation of a software routine should clarify what it is doing, as well as define the required and optional inputs and the available outputs. Helpful documentation is aimed at a knowledgeable but not necessarily expert reader who is new to this program, yet familiar with the MATLAB language.

Where possible, the documentation should be generated from the m-file with little effort. The document should have readability features that make it easier to read than simple unformatted comments. In the case of the MATLAB product, this means using the Publish feature.

Provide Well-Written Code

The best documentation for a program is clean code. Comments cannot justify poorly written code, nor can they make up for code lacking appropriate name choices, good layout, or an explicit logical structure. Such code should be rewritten.

Document Each Module Before or During Its Implementation

Development projects are rarely completed on schedule. If documentation is left for last, then it will get cut short. Such documentation is often too little, too late, or even absent. Writing some documentation early assures that it gets done, and this practice will probably reduce development time. Writing at least some of the documentation before coding will also encourage you to think more about the functionality and interfaces of the modules.

Solution verification is an important aspect of ensuring that a given simulation of a mathematical model is sufficiently accurate for the intended use. It relies on the use of consistent and convergent numerical algorithms as well as mistake-free codes; the two key items addressed in Part II of this book. If code verification studies have not been conducted, then even the most rigorous solution verification activities are not sufficient since there is no guarantee that the simulations will converge to the exact solution to the mathematical model. Just as code verification is a necessary prelude to solution verification, meaningful model validation assessments (Part IV) cannot be conducted until solution verification has been completed.

The main focus of solution verification is the estimation of the numerical errors that occur when a mathematical model is discretized and solved on a digital computer. While some of the strategies employed will be similar to those used for code verification, there is an important difference. In solution verification, the exact solution to the mathematical model is not known, and thus the numerical errors must now be estimated and not simply evaluated. In some cases, when these numerical errors can be estimated with a high degree of confidence, then they can be removed from the numerical solution (a process similar to that used for well-characterized bias errors in an experiment). More often, however, the numerical errors are estimated with significantly less certainty, and thus they will be classified as numerical uncertainties.

Solution verification addresses the question of whether a given simulation (i.e., numerical approximation) of a mathematical model is sufficiently accurate for its intended use. It includes not only the accuracy of the simulation for the case of interest, but also the accuracy of inputs to the code and any post-processing of the code results. Quantifying the numerical accuracy of scientific computing simulations is important for two primary reasons: as part of the quantification of the total uncertainty in a simulation prediction (Chapter 13) and for establishing the numerical accuracy of a simulation for model validation purposes (Chapter 12).

Most solution verification activities are focused on estimating the numerical errors in the simulation. This chapter addresses in detail round-off error, statistical sampling error, and iterative convergence error. These three numerical error sources should be sufficiently small so as not to impact the estimation of discretization error, which is discussed at length in Chapter 8. Discretization errors are those associated with the mesh resolution and quality as well as the time step chosen for unsteady problems. Round-off and discretization errors are always present in scientific computing simulations, while the presence of iterative and statistical sampling errors will depend on the application and the chosen numerical algorithms. This chapter concludes with a discussion of numerical errors and their relationship to uncertainties.

As we begin to address issues of validation in Part IV: Model validation and prediction (Chapters 10–13), the focus will be on whether the proper mathematical model has been chosen, where mathematical model refers to the governing partial differential or integral equations along with any auxiliary algebraic relations. Since the exact solutions to complex mathematical models are extremely rare, we generally use numerical solutions to the discretized equations as a surrogate for the exact solutions. Verification provides a framework for quantifying the numerical approximation errors in the discrete solution relative to the exact solution to the mathematical model. Since verification deals purely with issues of mathematics, no references to the actual behavior of real-world systems or experimental data will be found in Chapters 4 through 9.

Code verification ensures that the computer program (alternatively referred to as the computer code) is a faithful representation of the original mathematical model. It is accomplished by employing appropriate software engineering practices (Chapter 4), and by using order verification (Chapter 5) to ensure that there are no mistakes in the computer code or inconsistencies in the discrete algorithm. This part of the book dealing with code verification is completed by a discussion of exact solutions to mathematical models in Chapter 6. A key part of Chapter 6 is the Method of Manufactured Solutions (MMS), which is a powerful method for performing order verification studies on complex, nonlinear, coupled sets of partial differential or integral equations.

The primary focus of this chapter is on the use of exact solutions to mathematical models for code verification. Recall that, in some cases, software testing can be performed by simply running the code and comparing the results to the correct code output. However, in scientific computing, “correct” code output depends on the chosen spatial mesh, time step, iterative convergence tolerance, machine precision, etc. We are thus forced to rely on other less definitive methods for assessing code correctness. In Chapter 5, the order of accuracy test was argued to be the most rigorous approach for code verification. When the order of accuracy test fails, or when the formal order of accuracy has not been determined, then the less rigorous convergence test may be used. In either case, an exact solution to the underlying mathematical model is required. When used for code verification, the ability of this exact solution to exercise all terms in the mathematical model is more important than any physical realism of the solution. In fact, realistic exact solutions are often avoided for code verification due to the presence of singularities and/or discontinuities. Numerous examples will be given in this chapter of exact solutions and their use with the order verification test. The final example given in this chapter employs the less rigorous convergence test with benchmark numerical solutions. In addition to code verification applications, exact solutions to mathematical models are extremely valuable for evaluating the accuracy of numerical schemes, determining solution sensitivity to mesh quality and topology, evaluating the reliability of discretization error estimators, and evaluating solution adaptation schemes. For these secondary applications, physically realistic exact solutions are preferred (see Section 6.4).

Modeling and simulation are used in a myriad of ways in business and government. The range covers science, engineering and technology, industry, environmental regulations and safety, product and plant safety, financial investing, design of military systems, governmental planning, and many more. In all of these activities models are built that are mental constructs of how we believe the activity functions and how it is influenced by events or surroundings. All models are abstractions of the real activity that are based on many different types of approximation. These models are then programmed for execution on a digital computer, and the computer produces a simulation result. The simulation result may have high fidelity to the actual activity of interest, or it may be complete nonsense. The question is: how can we tell which is which? This book deals with various technical and procedural tools that can be used to assess the fidelity of modeling and simulation aspects of scientific computing. Our focus is on physical processes and systems in a broad range of the natural sciences and engineering.

The tools discussed here are primarily focused on mathematical models that are represented by differential and/or integral equations. Many of these mathematical models occur in physics, chemistry, astronomy, Earth sciences, and engineering, but they also occur in other fields of modeling and simulation. The topics addressed in this book are all related to the principles involved in assessing the credibility of the models and the simulation results. We do not deal with the specific details of modeling the physical process or system of interest, but with assessment procedures relating to the fidelity of the models and simulations. These procedures are typically described by the terms verification and validation.

In Chapter 1, Introduction, we briefly discussed how credibility is built in modeling and simulation (M&S). The four elements mentioned in that chapter were: quality of the analysts conducting the analysis, quality of the physics modeling, verification and validation activities, and uncertainty quantification and sensitivity analysis. The latter three elements are technical elements that can be assessed for completeness or maturity. Assessment of maturity is important to the staff conducting the modeling and simulation effort, but it is critically important for project managers and decision makers who use computational results as an element in their decision making. It is also important for internal or external review committees who are asked to provide recommendations on the credibility and soundness of computational analyses. This chapter deals with reviewing methods that have been developed for assessing similar activities, and then presents a newly developed technique reported in Oberkampf et al. (2007). This chapter is taken in large part from this reference.

Survey of maturity assessment procedures

Over the last decade, a number of researchers have investigated how to measure the maturity and credibility of software and hardware development processes and products. Probably the best-known procedure for measuring the maturity of software product development and business processes is the Capability Maturity Model Integration (CMMI). The CMMI is a successor to the Capability Maturity Model (CMM). Development of the CMM was initiated in 1987 to improve software quality. For an extensive discussion of the framework and methods for the CMMI, see West (2004); Ahern et al. (2005); Garcia and Turner (2006); and Chrissis et al. (2007).

This chapter discusses the fundamental concepts and terminology associated with verification and validation (V&V) of models and simulations. We begin with a brief history of the philosophical foundations so that the reader can better understand why there are a wide variety of views toward V&V principles and procedures. Various perspectives of V&V have also generated different formal definitions of the terms verification and validation in important communities. Although the terminology is moving toward convergence within some communities, there are still significant differences. The reader needs to be aware of these differences in terminology to help minimize confusion and unnecessary disagreements, as well as to anticipate possible difficulties in contractual obligations in business and government. We also discuss a number of important and closely related terms in modeling and simulation (M&S). Examples are predictive capability, calibration, certification, uncertainty, and error. We end the chapter with a discussion of a conceptual framework for integrating verification, validation, and predictive capability. Although there are different frameworks for integrating these concepts, the framework discussed here has proven very helpful in understanding how the various activities in scientific computing are related.

This chapter briefly sketches the historical beginnings of modeling and simulation (M&S). Although claiming the beginning of anything is simply a matter of convenience, we will start with the stunning invention of calculus. We then discuss how the steadily increasing performance and decreasing costs of computing have been another critical driver in advancing M&S. Contributors to the credibility of M&S are discussed, and the preliminary concepts of verification and validation are mentioned. We close the chapter with an outline of the book and suggest how the book might be used by students and professionals.

Historical and modern role of modeling and simulation

Historical role of modeling and simulation

For centuries, the primary method for designing an engineered system has been to improve the successful design of an existing system incrementally. During and after the system was built, it would be gradually tested in a number of ways. The first tests would usually be done during the building process in order to begin to understand the characteristics and responses of the new system. This new system was commonly a change in the old system's geometrical character, materials, fastening techniques, or assembly techniques, or a combination of all of these changes. If the system was intended to be used in some new environment such as a longer bridge span, a taller structure, or propelled at higher speeds, the system was always tested first in environments where the experience base already existed. Often, during the building and testing process, design or assembly weaknesses and flaws were discovered and modifications to the system were made. Sometimes a catastrophic failure of a monumental project would occur and the process would start over: occasionally after attending the funeral of the previous chief designer and his apprentices (DeCamp, 1995). In ancient times, chief designers understood the consequences of a major design failure; they had skin in the game.

The phrases modeling and simulation and computational simulation are becoming prevalent in a wide variety of technical, economic, governmental, and business activities (Schrage, 1999). Indeed, the phrases are becoming so common that one is even beginning to see them in the mass media. What do they mean? These phrases can have a wide variety of meanings depending on the field and the context. Here, we are concerned with the fields of the physical sciences and engineering. By examining the fundamentals of modeling and simulation (M&S) and scientific computing, our goal is to see the similarities in model formulation and computational issues across a wide range of physical systems. Our approach to scientific computing emphasizes the similarities that exist in mathematical form and structure of models in many technical disciplines. Then a framework, an overarching structure, is constructed for either attacking more detailed features of the system or for attacking more complex systems. Commonly, more complex systems involve coupling different types of physical phenomena, incorporation of additional elements from the system or the surroundings, and effects of human intervention in the system. Similarities in model formulation issues exist because many of the model properties are not determined by their physical nature, but by their mathematical structure, the interaction of the system with the surroundings, and the similarity in the nature of the system responses.

Many of the difficult issues that must be dealt with in verification, validation, and uncertainty quantification (VV&UQ) can be traced back to ambiguities and inconsistencies in the model formulation, the mapping of continuum mathematics models to discrete mathematics models, and vague or improper characterizations of uncertainties. This chapter deals with many of those issues by examining the fundamentals of M&S.

As has been discussed in a number of chapters, particularly Chapter 10, Model validation fundamentals, and Chapter 11, Design and execution of validation experiments, model accuracy assessment is the core issue of model validation. Our intent in model accuracy assessment is to critically and quantitatively determine the ability of a mathematical model and its embodiment in a computer code to simulate a well-characterized physical process. We, of course, are only interested in well-characterized physical processes that are useful for model validation. How critical and quantitative the model accuracy assessment is will depend on (a) how extensive the experimental data set is in exploring the important model input quantities that affect the system response quantities (SRQs) of interest; (b) how well characterized the important model input quantities are, based on measurements in the experiments; (c) how well characterized the experimental measurements and the model predictions of the SRQs of interest are; (d) whether the experimental measurements of the SRQs were available to the computational analyst before the model accuracy assessment was conducted; and (e) if the SRQs were available to the computational analysts, whether they were used for model updating or model calibration. This chapter will explore these difficult issues both conceptually and quantitatively.

We begin the chapter by discussing the fundamental elements of model accuracy assessment. As part of this discussion, we review traditional and recent methods for comparing model results and experimental measurements, and we explore the relationship between model accuracy assessment, model calibration, and model prediction. Beginning with the engineering society definitions of terms given in Chapter 2, Fundamental concepts and terminology, the perspective of this book is to segregate, as well as possible, each of these activities. There is, however, an alternative perspective in the published literature that believes all of these activities should be combined. We briefly review this alternative perspective and the associated approaches, and contrast these with approaches that segregate these activities.

This chapter will synthesize the key results from the previous chapters and incorporate them into modern predictive capability in scientific computing. This chapter, in contrast to all other chapters, does not stress the theme of assessment. Here we discuss the fundamental steps in conducting a nondeterministic analysis of a system of interest. With this discussion we show how verification and validation (V&V) can directly contribute to predictive capability.

The previously covered material and the new material are organized into six procedural steps to make a prediction:

1. identify all relevant sources of uncertainty,

2. characterize each source of uncertainty,

3. estimate numerical solution error in the system response quantities of interest,

4. estimate uncertainty in the system response quantities of interest,

5. conduct model updating,

6. conduct sensitivity analysis.

All of these steps, except step 3, are widely practiced in nondeterministic simulations and risk analysis. Step 3 is not commonly addressed for three reasons. First, in many simulations the numerical solution error is assumed to be small compared to other contributors to uncertainty. Sometimes this assumption is quantitatively justified, and sometimes it is simply posited with little or no evidence. Second, in some computationally intensive simulations, it is understood that the numerical solution error is important, and possibly even dominant, but it is argued that various modeling parameters can be adjusted to compensate for the numerical error. If the application of interest is sufficiently similar to the conditions for which experimental data are available, then it is claimed that the adjustable parameters can be used to match the existing data and thereby make reasonable predictions. Third, even if the numerical error is estimated, and it is not small relative to other uncertainties, there are no generally accepted procedures for including its effect on the system response quantities (SRQs) of interest.

In scientific computing, the goal of code verification is to ensure that the code is a faithful representation of the underlying mathematical model. This mathematical model generally takes the form of partial differential or integral equations along with associated initial condition, boundary conditions, and auxiliary relationships. Code verification thus addresses both the correctness of the chosen numerical algorithm and the correctness of the instantiation of that algorithm into written source code, i.e., ensuring there are no coding mistakes or “bugs.”

A computer program, referred to here simply as a code, is a collection of instructions for a computer written in a programming language. As discussed in Chapter 4, in the software engineering community code verification is called software verification and is comprised of software tests which ensure that the software meets the stated requirements. When conducting system-level testing of non-scientific software, in many cases it is possible to exactly determine the correct code output for a set of given code inputs. However, in scientific computing, the code output depends on the numerical algorithm, the spatial mesh, the time step, the iterative tolerance, and the number of digits of precision used in the computations. Due to these factors, it is not possible to know the correct code output (i.e., numerical solution) a priori. The developer of scientific computing software is thus faced with the difficult challenge of determining appropriate system-level software tests.