Good software gets the immediate job done. But great software, written with a consistent and readable style, is predictable, robust, maintainable, supportable, and extensible. A few general principles provide the foundation for great software.

Avoid Causing Confusion

Avoid doing things that would be an unpleasant surprise to other software developers. The interfaces and the behavior exhibited by your software must be predictable and consistent. If they are not, then the documentation must clearly identify and justify any unusual instances of use or behavior.

To minimize the chances that anyone would encounter something surprising in your software, you should emphasize the following characteristics in its design, implementation, packaging, and documentation:

1. Simplicity Meet the expectations of your users with simple functions, classes, and methods.

2. Clarity Ensure that each variable, function, class, and method has a clear purpose.

3. Completeness Provide at least the minimum functionality that any reasonable user would expect to find and use.

4. Consistency Design similar entities with a similar look and behavior. Create and apply consistent standards whenever possible.

5. Robustness Provide predictable, documented behavior in response to errors and exceptions. Do not hide errors, and do not force users to detect errors.

Avoid Throw-Away Code

Apply these rules to any code you write, not just code destined for production. All too often, some piece of prototype or experimental code will make its way into a finished product. Even if your code never makes it into production, someone else may still have to read it.

General

There are large variations in the amount and character of documentation for software development projects. MATLAB-specific style issues are concentrated in the internal documentation intimately associated with the code files, particularly comments and reference pages. Documentation of a software routine should clarify what it is doing, as well as define the required and optional inputs and the available outputs. Helpful documentation is aimed at a knowledgeable but not necessarily expert reader who is new to this program, yet familiar with the MATLAB language.

Where possible, the documentation should be generated from the m-file with little effort. The document should have readability features that make it easier to read than simple unformatted comments. In the case of the MATLAB product, this means using the Publish feature.

Provide Well-Written Code

The best documentation for a program is clean code. Comments cannot justify poorly written code, nor can they make up for code lacking appropriate name choices, good layout, or an explicit logical structure. Such code should be rewritten.

Document Each Module Before or During Its Implementation

Development projects are rarely completed on schedule. If documentation is left for last, then it will get cut short. Such documentation is often too little, too late, or even absent. Writing some documentation early assures that it gets done, and this practice will probably reduce development time. Writing at least some of the documentation before coding will also encourage you to think more about the functionality and interfaces of the modules.

Solution verification is an important aspect of ensuring that a given simulation of a mathematical model is sufficiently accurate for the intended use. It relies on the use of consistent and convergent numerical algorithms as well as mistake-free codes; the two key items addressed in Part II of this book. If code verification studies have not been conducted, then even the most rigorous solution verification activities are not sufficient since there is no guarantee that the simulations will converge to the exact solution to the mathematical model. Just as code verification is a necessary prelude to solution verification, meaningful model validation assessments (Part IV) cannot be conducted until solution verification has been completed.

The main focus of solution verification is the estimation of the numerical errors that occur when a mathematical model is discretized and solved on a digital computer. While some of the strategies employed will be similar to those used for code verification, there is an important difference. In solution verification, the exact solution to the mathematical model is not known, and thus the numerical errors must now be estimated and not simply evaluated. In some cases, when these numerical errors can be estimated with a high degree of confidence, then they can be removed from the numerical solution (a process similar to that used for well-characterized bias errors in an experiment). More often, however, the numerical errors are estimated with significantly less certainty, and thus they will be classified as numerical uncertainties.

Solution verification addresses the question of whether a given simulation (i.e., numerical approximation) of a mathematical model is sufficiently accurate for its intended use. It includes not only the accuracy of the simulation for the case of interest, but also the accuracy of inputs to the code and any post-processing of the code results. Quantifying the numerical accuracy of scientific computing simulations is important for two primary reasons: as part of the quantification of the total uncertainty in a simulation prediction (Chapter 13) and for establishing the numerical accuracy of a simulation for model validation purposes (Chapter 12).

Most solution verification activities are focused on estimating the numerical errors in the simulation. This chapter addresses in detail round-off error, statistical sampling error, and iterative convergence error. These three numerical error sources should be sufficiently small so as not to impact the estimation of discretization error, which is discussed at length in Chapter 8. Discretization errors are those associated with the mesh resolution and quality as well as the time step chosen for unsteady problems. Round-off and discretization errors are always present in scientific computing simulations, while the presence of iterative and statistical sampling errors will depend on the application and the chosen numerical algorithms. This chapter concludes with a discussion of numerical errors and their relationship to uncertainties.

As we begin to address issues of validation in Part IV: Model validation and prediction (Chapters 10–13), the focus will be on whether the proper mathematical model has been chosen, where mathematical model refers to the governing partial differential or integral equations along with any auxiliary algebraic relations. Since the exact solutions to complex mathematical models are extremely rare, we generally use numerical solutions to the discretized equations as a surrogate for the exact solutions. Verification provides a framework for quantifying the numerical approximation errors in the discrete solution relative to the exact solution to the mathematical model. Since verification deals purely with issues of mathematics, no references to the actual behavior of real-world systems or experimental data will be found in Chapters 4 through 9.

Code verification ensures that the computer program (alternatively referred to as the computer code) is a faithful representation of the original mathematical model. It is accomplished by employing appropriate software engineering practices (Chapter 4), and by using order verification (Chapter 5) to ensure that there are no mistakes in the computer code or inconsistencies in the discrete algorithm. This part of the book dealing with code verification is completed by a discussion of exact solutions to mathematical models in Chapter 6. A key part of Chapter 6 is the Method of Manufactured Solutions (MMS), which is a powerful method for performing order verification studies on complex, nonlinear, coupled sets of partial differential or integral equations.

The primary focus of this chapter is on the use of exact solutions to mathematical models for code verification. Recall that, in some cases, software testing can be performed by simply running the code and comparing the results to the correct code output. However, in scientific computing, “correct” code output depends on the chosen spatial mesh, time step, iterative convergence tolerance, machine precision, etc. We are thus forced to rely on other less definitive methods for assessing code correctness. In Chapter 5, the order of accuracy test was argued to be the most rigorous approach for code verification. When the order of accuracy test fails, or when the formal order of accuracy has not been determined, then the less rigorous convergence test may be used. In either case, an exact solution to the underlying mathematical model is required. When used for code verification, the ability of this exact solution to exercise all terms in the mathematical model is more important than any physical realism of the solution. In fact, realistic exact solutions are often avoided for code verification due to the presence of singularities and/or discontinuities. Numerous examples will be given in this chapter of exact solutions and their use with the order verification test. The final example given in this chapter employs the less rigorous convergence test with benchmark numerical solutions. In addition to code verification applications, exact solutions to mathematical models are extremely valuable for evaluating the accuracy of numerical schemes, determining solution sensitivity to mesh quality and topology, evaluating the reliability of discretization error estimators, and evaluating solution adaptation schemes. For these secondary applications, physically realistic exact solutions are preferred (see Section 6.4).

Modeling and simulation are used in a myriad of ways in business and government. The range covers science, engineering and technology, industry, environmental regulations and safety, product and plant safety, financial investing, design of military systems, governmental planning, and many more. In all of these activities models are built that are mental constructs of how we believe the activity functions and how it is influenced by events or surroundings. All models are abstractions of the real activity that are based on many different types of approximation. These models are then programmed for execution on a digital computer, and the computer produces a simulation result. The simulation result may have high fidelity to the actual activity of interest, or it may be complete nonsense. The question is: how can we tell which is which? This book deals with various technical and procedural tools that can be used to assess the fidelity of modeling and simulation aspects of scientific computing. Our focus is on physical processes and systems in a broad range of the natural sciences and engineering.

The tools discussed here are primarily focused on mathematical models that are represented by differential and/or integral equations. Many of these mathematical models occur in physics, chemistry, astronomy, Earth sciences, and engineering, but they also occur in other fields of modeling and simulation. The topics addressed in this book are all related to the principles involved in assessing the credibility of the models and the simulation results. We do not deal with the specific details of modeling the physical process or system of interest, but with assessment procedures relating to the fidelity of the models and simulations. These procedures are typically described by the terms verification and validation.

In Chapter 1, Introduction, we briefly discussed how credibility is built in modeling and simulation (M&S). The four elements mentioned in that chapter were: quality of the analysts conducting the analysis, quality of the physics modeling, verification and validation activities, and uncertainty quantification and sensitivity analysis. The latter three elements are technical elements that can be assessed for completeness or maturity. Assessment of maturity is important to the staff conducting the modeling and simulation effort, but it is critically important for project managers and decision makers who use computational results as an element in their decision making. It is also important for internal or external review committees who are asked to provide recommendations on the credibility and soundness of computational analyses. This chapter deals with reviewing methods that have been developed for assessing similar activities, and then presents a newly developed technique reported in Oberkampf et al. (2007). This chapter is taken in large part from this reference.

Survey of maturity assessment procedures

Over the last decade, a number of researchers have investigated how to measure the maturity and credibility of software and hardware development processes and products. Probably the best-known procedure for measuring the maturity of software product development and business processes is the Capability Maturity Model Integration (CMMI). The CMMI is a successor to the Capability Maturity Model (CMM). Development of the CMM was initiated in 1987 to improve software quality. For an extensive discussion of the framework and methods for the CMMI, see West (2004); Ahern et al. (2005); Garcia and Turner (2006); and Chrissis et al. (2007).

This chapter discusses the fundamental concepts and terminology associated with verification and validation (V&V) of models and simulations. We begin with a brief history of the philosophical foundations so that the reader can better understand why there are a wide variety of views toward V&V principles and procedures. Various perspectives of V&V have also generated different formal definitions of the terms verification and validation in important communities. Although the terminology is moving toward convergence within some communities, there are still significant differences. The reader needs to be aware of these differences in terminology to help minimize confusion and unnecessary disagreements, as well as to anticipate possible difficulties in contractual obligations in business and government. We also discuss a number of important and closely related terms in modeling and simulation (M&S). Examples are predictive capability, calibration, certification, uncertainty, and error. We end the chapter with a discussion of a conceptual framework for integrating verification, validation, and predictive capability. Although there are different frameworks for integrating these concepts, the framework discussed here has proven very helpful in understanding how the various activities in scientific computing are related.

This chapter briefly sketches the historical beginnings of modeling and simulation (M&S). Although claiming the beginning of anything is simply a matter of convenience, we will start with the stunning invention of calculus. We then discuss how the steadily increasing performance and decreasing costs of computing have been another critical driver in advancing M&S. Contributors to the credibility of M&S are discussed, and the preliminary concepts of verification and validation are mentioned. We close the chapter with an outline of the book and suggest how the book might be used by students and professionals.

Historical and modern role of modeling and simulation

Historical role of modeling and simulation

For centuries, the primary method for designing an engineered system has been to improve the successful design of an existing system incrementally. During and after the system was built, it would be gradually tested in a number of ways. The first tests would usually be done during the building process in order to begin to understand the characteristics and responses of the new system. This new system was commonly a change in the old system's geometrical character, materials, fastening techniques, or assembly techniques, or a combination of all of these changes. If the system was intended to be used in some new environment such as a longer bridge span, a taller structure, or propelled at higher speeds, the system was always tested first in environments where the experience base already existed. Often, during the building and testing process, design or assembly weaknesses and flaws were discovered and modifications to the system were made. Sometimes a catastrophic failure of a monumental project would occur and the process would start over: occasionally after attending the funeral of the previous chief designer and his apprentices (DeCamp, 1995). In ancient times, chief designers understood the consequences of a major design failure; they had skin in the game.