Download Error Analysis in Numerical Analysis: Approximations and Conditioning and more Study notes Mathematical Methods for Numerical Analysis and Optimization in PDF only on Docsity!
Jim Lambers Math 105A Summer Session I 2003- Lecture 3 Notes
These notes correspond to Section 1.3 in the text.
Approximations in Numerical Analysis
Mathematical problems arising from scientific applications present a wide variety of difficulties that prevent us from solving them exactly. This has led to an equally wide variety of techniques for computing approximations to quantities occurring in such problems in order to obtain approximate solutions. In this lecture, we will describe the types of approximations that can be made, and learn some basic techniques for analyzing the accuracy of these approximations.
Sources of Approximation
Suppose that we are attempting to solve a particular instance of a problem arising from a mathe- matical model of a scientific application. We say that such a problem is well-posed if it meets the following criteria:
- The problem has a unique solution.
- A small perturbation in the problem data results in a small perturbation in the solution; i.e., the solution depends continuously on the data.
By the first condition, the process of solving a well-posed problem can be seen to be equivalent to the evaluation of some function f at some known value x, where x represents the problem data. Since, in many cases, knowledge of the function f is limited, the task of computing f (x) can be viewed, at least conceptually, as the execution of some (possibly infinite) sequence of steps that solves the underlying problem for the data x. The goal in numerical analysis is to develop a finite sequence of steps, i.e., an algorithm, for computing an approximation to the value f (x). There are two general types of error that occur in the process of computing this approximation to f (x):
- data error is the error in the data x. In reality, numerical analysis involves solving a problem with approximate data ˆx. The exact data is often unavailable because it must be obtained by measurements or other computations that fail to be exact due to limited precision. In addition, data may be altered in order to simplify the solution process.
- computational error refers to the error that occurs when attempting to compute f (ˆx). Ef- fectively, we must approximate f (ˆx) by the quantity fˆ (ˆx), where fˆ is a function that ap- proximates f. This approximation may be the result of truncation, which occurs when it is not possible to evaluate f exactly using a finite sequence of steps, and therefore a finite sequence that evaluates f approximately must be used instead. This particular source of computational error will be discussed in this lecture. Another source of computational error is roundoff error, which was discussed in the previous lecture.
Basics of Error Analysis
Intuitively, it is not difficult to conclude that any scientific computation can include several ap- proximations, each of which introduces error in the computed solution. Therefore, it is necessary to understand the effects of these approximations on accuracy. The study of these effects is known as error analysis. Error analysis will be a recurring theme in this course. In this lecture, we will introduce some basic concepts that will play a role in error analyses of specific algorithms in later lectures.
Forward Error and Backward Error
Suppose that we compute an approximation ˆy = fˆ (x) of the value y = f (x) for a given function f and given problem data x. Before we can analyze the accuracy of this approximation, we must have a precisely defined notion of error in such an approximation. We now provide this precise definition.
Definition (Forward Error) Let x be a real number and let f : R → R be a function. If ˆy is a real number that is an approximation to y = f (x), then the forward error in yˆ is the difference ∆y = ˆy − y. If y 6 = 0, then the relative forward error in ˆy is defined by
∆y y
yˆ − y y
Clearly, our primary goal in error analysis is to obtain an estimate of the forward error ∆y. Un- fortunately, it can be difficult to obtain this estimate directly. An alternative approach is to instead view the computed value ˆy as the exact solution of a problem with modified data; i.e., ˆy = f (ˆx) where ˆx is a perturbation of x.
Definition (Backward Error) Let x be a real number and let f : R → R be a function. Suppose that the real number yˆ is an approximation to y = f (x), and that ˆy is in the range of f ; that is, y ˆ = f (ˆx) for some real number xˆ. Then, the quantity ∆x = ˆx − x is the backward error in yˆ. If x 6 = 0, then the relative forward error in yˆ is defined by
∆x x
ˆx − x x
that end, we assume, for simplicity, that f : R → R is differentiable and obtain
κrel =
|x∆y| |y∆x|
= |x(f (x + ∆x) − f (x))| |f (x)∆x|
≈ |xf ′(x)∆x| |f (x)∆x|
≈
∣∣^ xf^
′(x) f (x)
Therefore, if we can estimate the backward error ∆x, and if we can bound f and f ′^ near x, we can then bound the condition number and obtain an estimate of the relative forward error. Of course, the condition number is undefined if the exact value f (x) is zero. In this case, we can instead use the absolute condition number. Using the same approach as before, the absolute condition number can be estimated using the derivative of f. Specifically, we have κabs ≈ |f ′(x)|.
Conditioning, Stability and Accuracy
Determining the condition, or sensitivity, of a problem is an important task in the error analysis of an algorithm designed to solve the problem, but it does not provide sufficient information to determine whether an algorithm will yield an accurate approximate solution. Recall that the condition number of a function f depends on, among other things, the absolute forward error f (ˆx) − f (x). However, an algorithm for evaluating f (x) actually evaluates a function fˆ that approximates f , producing an approximation ˆy = fˆ (x) to the exact solution y = f (x). In our definition of backward error, we have assumed that fˆ (x) = f (ˆx) for some ˆx that is close to x; i.e., our approximate solution to the original problem is the exact solution to a “nearby” problem. This assumption has allowed us to define the condition number of f independently of any approximation fˆ. This independence is necessary, because the sensitivity of a problem depends solely on the problem itself and not any algorithm that may be used to approximately solve it. Is it always reasonable to assume that any approximate solution is the exact solution to a nearby problem? Unfortunately, it is not. It is possible that an algorithm that yields an accurate approximation for given data may be unreasonably sensitive to perturbations in that data. This leads to the concept of a stable algorithm: an algorithm applied to a given problem with given data x is said to be stable if it computes an approximate solution that is the exact solution to the same problem with data ˆx, where ˆx is a small perturbation of x. It can be shown that if a problem is well-conditioned, and if we have a stable algorithm for solving it, then the computed solution can be considered accurate, in the sense that the relative error in the computed solution is small. On the other hand, a stable algorithm applied to an ill-conditioned problem cannot be expected to produce an accurate solution.
Convergence
Many algorithms in numerical analysis are iterative methods that produce a sequence {αn} of ap- proximate solutions which, ideally, converges to a limit α that is the exact solution as n approaches ∞. Because we can only perform a finite number of iterations, we cannot obtain the exact solution, and we have introduced computational error. If our iterative method is properly designed, then this computational error will approach zero as n approaches ∞. However, it is important that we obtain a sufficiently accurate approximate solution using as few computations as possible. Therefore, it is not practical to simply perform enough iterations so that the computational error is determined to be sufficiently small, because it is possible that another method may yield comparable accuracy with less computational effort. The total computational effort of an iterative method depends on both the effort per iteration and the number of iterations performed. Therefore, in order to determine the amount of compu- tation that is needed to attain a given accuracy, we must be able to measure the error in αn as a function of n. The more rapidly this function approaches zero as n approaches ∞, the more rapidly the sequence of approximations {αn} converges to the exact solution α, and as a result, fewer iterations are needed to achieve a desired accuracy. We now introduce some terminology that will aid in the discussion of the convergence behavior of iterative methods.
Definition (Big-O Notation) Let f and g be two functions defined on a domain D ⊆ R that is not bounded above. We write that f (n) = O(g(n)) if there exists a positive constant c such that
|f (n)| ≤ c|g(n)|, n ≥ n 0 ,
for some n 0 ∈ D.
As sequences are functions defined on N, the domain of the natural numbers, we can apply big-O notation to sequences. Therefore, this notation is useful to describe the rate at which a sequence of computations converges to a limit.
Definition (Rate of Convergence) Let {αn}∞ n=1 and {βn}∞ n=1 be sequences that satisfy
lim n→∞ αn = α, lim n→∞ βn = 0,
where α is a real number. We say that {αn} converges to α with rate of convergence O(βn) if αn − α = O(βn).
We say that an iterative method converges rapidly, in some sense, if it produces a sequence of approximate solutions whose rate of convergence is O(βn), where the terms of the sequence βn approach zero rapidly as n approaches ∞. Intuitively, if two iterative methods for solving the same problem perform a comparable amount of computation during each iteration, but one method exhibits a faster rate of convergence, then that method should be used because it will require less overall computational effort to obtain an approximate solution that is sufficiently accurate.
