In this chapter, we establish the mathematical foundation for hard computing optimization algorithms. We look at the classical optimization approaches and extend our discussion to include iterative methods, which hold a special role in machine learning. In particular, we review the gradient decent method, Newton’s method, the conjugate gradient method and the quasi-Newton’s method. Along with the discussion of these optimization methods, implementation using Matlab script as well as considerations for use in neural network training algorithms are provided. Finally, the Levenberg-Marquardt method is introduced, discussed, and implemented in Matlab script to compare its functioning with the other four iterative algorithms introduced in this chapter.

Chapter 5 starts out with a physics motivation, as well as a mathematical statement of the problems that will be tackled in later sections. Several methods are introduced to solve a single nonlinear equation in one variable: fixed-point iteration, the bisection method, Newton’s method, the secant method, and Ridders’ method. After providing some advice about advantages and disadvantages of each approach, the text then studies how to find zeros of polynomials, employing two different techniques. The sophistication is then increased, by tackling systems of nonlinear equations and examining the corresponding challenges; in addition to Newton’s method, the text derives the equations behind Broyden’s method. A related subject is then broached, minimization in one or several dimensions; this includes the gradient-descent method, as well as detailed analysis of critical points; the second edition includes extensive new material on derivative-free optimization (golden-section search and Powell’s method).The chapter is rounded out by a physics project, the extremization of the action in classical mechanics, and a problem set. The physics project shows Hamilton’s principle in... action, translated into a multidimensional minimization problem.

This chapter serves two purposes: it introduces several essential concepts of linear and nonlinear functional analysis that will be used in subsequent chapters and, as an illustration of them, studies the problem of unconstrained minimization of a convex functional. All the necessary notions of existence, uniqueness, and optimality conditions are presented and analyzed. Preconditioned gradient descent methods for strongly convex, locally Lipschitz smooth objectives in infinite dimensions are then presented and analyzed. A general framework to show linear convergence in this setting is then presented. The preconditioned steepest descent with exact and approximate line searches are then analyzed using the same framework. Finally, the application of Newton’s method to the Euler equations is discussed. The local convergence is shown, and how to achieve global convergence is briefly discussed.

This chpater is dedicated to the solution of nonlinear systems of equations, that is finding roots of functions. We begin with a classification of roots into simple and non-simple, and a few words about their stability. Then we begin with some of the simplest methods for root finding: bisection, false position, fixed point iterations, and its variants. For all these schemes, we provide sufficient conditions for them to be well defined and convergent. A detailed analysis of Newton’s method, and its variants (collectively known as quasi-Newton methods), in one dimension is then presented. We show sufficient conditions for its local and global quadratic convergence,as well as how to proceed in the case of non simple roots. Then we present Newton’s method in several dimensions, and show its local quadratic convergence, including the celebrated Kantorovich’s theorem.

Mathematical background and formulation of numerical minimization process are described in terms of gradient-based methods, whose ingredients include gradient, Hessian, directional derivatives, optimality conditions for minimization, Hessian eigensystem, conjugate number of Hessian, and conjugate vectors. Various minimization algorithms, such as the steepest descent method, Newton’s method, conjugate gradient method, and quasi-Newton’s method, are introduced along with practical examples.