Download Numerical Differentiation - Numerical Analysis - Lecture Notes | Math 105 and more Study notes Mathematical Methods for Numerical Analysis and Optimization in PDF only on Docsity!
Jim Lambers Math 105A Summer Session I 2003- Lecture 11 Notes
These notes correspond to Sections 4.1 and 4.2 in the text.
Numerical Differentiation
We now discuss the other fundamental problem from calculus that frequently arises in scientific applications, the problem of computing the derivative of a given function f (x).
Finite Difference Approximations
Recall that the derivative of f (x) at a point x 0 , denoted f ′(x 0 ), is defined by
f ′(x 0 ) = lim h→ 0
f (x 0 + h) − f (x 0 ) h
This definition suggests a method for approximating f ′(x 0 ). If we choose h to be a small positive constant, then
f ′(x 0 ) ≈ f (x 0 + h) − f (x 0 ) h
This approximation is called the forward difference formula. To estimate the accuracy of this approximation, we note that if f ′′(x) exists on [x 0 , x 0 + h], then, by Taylor’s Theorem, f (x 0 + h) = f (x 0 ) + f ′(x 0 )h + f ′′(ξ)h^2 / 2 , where ξ ∈ [x 0 , x 0 + h]. Solving for f ′(x 0 ), we obtain
f ′(x 0 ) = f (x 0 + h) − f (x 0 ) h
f ′′(ξ) 2
h,
so the error in the forward difference formula is O(h). We say that this formula is first-order accurate. The forward-difference formula is called a finite difference approximation to f ′(x 0 ), because it approximates f ′(x) using values of f (x) at points that have a small, but finite, distance between them, as opposed to the definition of the derivative, that takes a limit and therefore computes the derivative using an “infinitely small” value of h. The forward-difference formula, however, is just one example of a finite difference approximation. If we replace h by −h in the forward-difference formula, where h is still positive, we obtain the backward-difference formula
f ′(x 0 ) ≈
f (x 0 ) − f (x 0 − h) h
Like the forward-difference formula, the backward difference formula is first-order accurate.
If we average these two approximations, we obtain the centered-difference formula
f ′(x 0 ) ≈
f (x 0 + h) − f (x 0 − h) 2 h
To determine the accuracy of this approximation, we assume that f ′′′(x) exists on the interval [x 0 − h, x 0 + h], and then apply Taylor’s Theorem again to obtain
f (x 0 + h) = f (x 0 ) + f ′(x 0 )h + f ′′(x 0 ) 2
h^2 + f ′′′(ξ+) 6
h^3 ,
f (x 0 − h) = f (x 0 ) − f ′(x 0 )h + f ′′(x 0 ) 2
h^2 − f ′′′(ξ−) 6
h^3 ,
where ξ+ ∈ [x 0 , x 0 + h] and ξ− ∈ [x 0 − h, x 0 ]. Subtracting the second equation from the first and solving for f ′(x 0 ) yields
f ′(x 0 ) = f (x 0 + h) − f (x 0 − h) 2 h
f ′′′(ξ+) + f ′′′(ξ−) 12
h^2.
By the Intermediate Value Theorem, f ′′′(x) must assume every value between f ′′′(ξ−) and f ′′′(ξ+) on the interval (ξ−, ξ+), including the average of these two values. Therefore, we can simplify this equation to
f ′(x 0 ) =
f (x 0 + h) − f (x 0 − h) 2 h
f ′′′(ξ) 6 h^2 ,
where ξ ∈ [x 0 −h, x 0 +h]. We conclude that the centered-difference formula is second-order accurate. This is due to the cancellation of the terms involving f ′′(x 0 ). While Taylor’s Theorem can be used to derive formulas with higher-order accuracy simply by evaluating f (x) at more points, this process can be tedious. An alternative approach is to compute the derivative of the interpolating polynomial that fits f (x) at these points. Specifically, suppose we want to compute the derivative at a point x 0 using the data
(x−j , y−j ),... , (x− 1 , y− 1 ), (x 0 , y 0 ), (x 1 , y 1 ),... , (xk, yk),
where j and k are known nonnegative integers, x−j < x−j+1 < · · · < xk− 1 < xk, and yi = f (xi) for i = −j,... , k. Then, a finite difference formula for f ′(x 0 ) can be obtained by analytically computing the derivatives of the Lagrange polynomials {Ln,i(x)}ki=−j for these points, where n = j + k + 1, and the values of these derivatives at x 0 are the proper weights for the function values y−j ,... , yk. If f (x) is n + 1 times continuously differentiable on [x−j , xk], then we obtain an approximation of the form
f ′(x 0 ) =
∑^ k
i=−j
yiL′ n,i(x 0 ) + f (n+1)(ξ) (n + 1)!
∏^ k
i=−j,i 6 =
(x 0 − xi),
where ξ ∈ [x−j , xk].
Automatic Differentiation
If the function f (x) can be represented as a sequence of computations, each consisting of the evaluation of some differentiable function with differentiable arguments, then a similar sequence of computations representing f ′(x) can be obtained using automatic differentiation. Suppose that the evaluation of f (x) consists of the evaluation of simpler differentiable functions t 1 ,... , tn, where t 1 only depends on x, and tj , for j > 1, depends on x and t 1 ,... , tj− 1 , with tn(x, t 1 ,... , tn− 1 ) representing the desired value f (x). Automatic differentiation proceeds by com- puting the derivatives of each function tj , j = 1,... , n, to obtain a new sequence of functions t′ j , j = 1,... , n, where t′ j depends on t 1 ,... , tj and t′ 1 ,... , t′ j− 1 , and f ′(x) = t′ n(x, t 1 ,... , tn, t′ 1 ,... , t′ n). This process, while conceptually simple, can be very difficult to implement since much opti- mization of the generated derivative is necessary. Fortunately, techniques used in compiler design are very useful in building a practical automatic differentiation procedure. An additional benefit of automatic differentiation is that its analysis of f (x) can be used to estimate its sensitivity to perturbations.
Richardson Extrapolation
We have seen that the accuracy of methods for computing derivatives of a function f (x) depends on the spacing between points at which f is evaluated, and that the approximation tends to the exact value as this spacing tends to 0. Suppose that a uniform spacing h is used. We denote by F (h) the approximation computed using the spacing h, from which it follows that the exact value is given by F (0). Let p be the order of accuracy in our approximation; that is,
F (h) = a 0 + a 1 hp^ + O(hr), r > p,
where a 0 is the exact value F (0). Then, if we choose a value for h and compute F (h) and F (h/q) for some positive integer q, then we can neglect the O(hr) terms and solve a system of two equations for the unknowns a 0 and a 1 , thus obtaining an approximation that is rth order accurate. If we can describe the error in this approximation in the same way that we can describe the error in our original approximation F (h), we can repeat this process to obtain an approximation that is even more accurate. This process of extrapolating from F (h) and F (h/q) to approximate F (0) with a higher order of accuracy is called Richardson extrapolation. In a sense, Richardson extrapolation is similar in spirit to Aitken’s ∆^2 method, as both methods use assumptions about the convergence of a sequence of approximations to “solve” for the exact solution, resulting in a more accurate method of computing approximations.
