Download Notes: Rate of Convergence and more Summaries Literature in PDF only on Docsity!
Notes: Rate of Convergence
Suppose we have a sequence of real numbers that converge to some point x:
x 0 , x 1 , x 2 ,... → x
Is there a way to talk about how fast the numbers are converging to x?
Definition: If
nlim→∞
|xn+1 − x| |xn − x|
= λ < ∞
then the sequence converges linearly to x. The constant λ > 0 is called the asymptotic error.
Definition: If
nlim→∞
|xn+1 − x| |xn − x|^2
= λ < ∞
(where λ > 0) then the sequence converges quadratically to x.
Definition: If
nlim→∞
|xn+1 − x| |xn − x|α^
= λ < ∞
then the sequence converges to x of order α. This is the definition we use to actually compute the rate of convergence- Typically, this means that we need to see if α = 1 or 2. Examples:
- Let xn = (^) n^1 k for some fixed k > 0. Then we know this sequence converges to 0. Compute the rate of convergence:
nlim→∞
|xn+1| |xn|α^
= limn→∞
1 (n+1)k 1 nαk
= limn→∞
( nα n + 1
)k
We get convergence for α = 1, but not α = 2, so this sequence converges linearly to 0. This is interesting- Sequences like the following:
xn =
n
, xn =
n^2
, xn =
n^3
all converge linearly to zero.
- Let xn = 10−^2
n (which converges to 0). Compute the rate of convergence:
nlim→∞
|xn+1| |xn|α^
= lim n→∞
10 −^2
n+
10 −^2 n·α It’s easy to see that if α = 2, we get convergence. In fact, using this as a template sequence, we can see how fast a quadratically convergent sequence moves. Compute the terms of the sequence:
10 −^2 , 10 −^4 , 10 −^8 , 10 −^16 , 10 −^32 , 10 −^64
so that after just 6 steps, we’re already past machine . Algorithms that are quadratically convergent are highly prized!
0.1 Fixed Point Iteration
Now let’s analyze the fixed point algorithm, xn+1 = f (xn) with fixed point r. We will see below that the key to the speed of convergence will be f ′(r).
Theorem (Convergence of Fixed Point Iteration): Let f be continuous on [a, b] and f ′ be continuous on (a, b). Furthermore, assume there exists k < 1 so that |f ′(x)| ≤ k for all x in (a, b).
- If f ′(r) 6 = 0, the sequence converges linearly to the fixed point.
- If f ′(r) = 0, the sequence converges at least quadratically to the fixed point (this is sometimes called superconvergence in the dynamical systems literature).
Proof: This is more than we did in class- We will also prove that the sequence xn → r, which we do first. The following computations show this (they come from a repeated application of the Mean Value Theorem):
|x 1 − r| = |f (x 0 ) − f (r)| ≤ |f ′(x∗)||x 0 − r| ≤ k|x 0 − r|
where x∗ is between x 0 and r. Continuing, we see that:
|xn − r| ≤ |f ′(x∗)||xn− 1 − r| ≤ k|xn− 1 − r| ≤ kn|x 0 − r|
with 0 ≤ k < 1, so that kn^ → 0 as n → ∞. Therefore, |xn − r| → 0 as n → ∞. An important note: While x∗^ depended on xn− 1 , the value of k does not depend on n- The assumption that |f ′(x)| ≤ k < 1 was important for this step. Now we show that the sequence converges linearly if f ′(r) 6 = 0:
nlim→∞
|xn+1 − r| |xn − r|
= lim n→∞
|f (xn) − f (r)| |xn − r|
= lim n→∞ |f ′(cn)|
where cn is between xn and r. Note that, because xn → r, then cn → r as well. Because f ′^ is continuous on our interval that contains all the cn’s and r,
nlim→∞ |f^ ′(cn)|^ =
∣∣ ∣∣f ′
( n^ lim→∞ cn
)∣∣ ∣∣ = |f ′(r)|
To get an idea about why the second case should give quadratic convergence, consider the Taylor expansion of f about the fixed point r:
f (x) = f (r) + f ′(r)(x − r) +
f ′′(c)(x − r)^2
where c is in the interval between x and r. Now, if f ′(r) = 0, and rearranging terms, we see that:
f (x) − f (r) (x − r)^2
f ′′(x∗) 2
As you might expect, the answer to the first question is not as exact as it was for Fixed Point Iteration: Theorem (Convergence of Newton’s Method): Let g be twice continuously differentiable on the interval (a, b). If r ∈ (a, b) such that g(r) = 0 and g′(r) 6 = 0, then there exists δ > 0 such that Newton’s Method will converge if started in the interval [r − δ, r + δ]. In this case, the sequence converges quadratically.
We make one observation to begin: Newton’s Method is a form of Fixed Point iteration:
xn+1 = F (xn) where F (x) = x −
g(x) g ′(x)
and the convergence of fixed point iteration depended on the derivative of F :
F ′(x) = 1 −
g ′(x)g ′(x) − g(x)g ′′(x) (g ′(x))^2
g(x)g ′′(x) (g ′(x))^2
we also see that F ′(r) = 0. To be able to use the Fixed Point Theorem (Item 3 above), we would need to show that F satisfies those conditions- Namely, that there is an interval (a, b) about r where F ′^ is continuous and |F ′(x)| < 1. Let’s see how our assumptions create the appropriate δ so that we guarantee the convergence of Newton’s Method. Here are some observations:
- (Continuity of F and F ′) F will be continuous if g and g′^ are continuous, and g′(x) 6 = 0. Now we have no problem with g- It is continuous on (a, b). Since g′(r) 6 = 0, and g′^ is continuous, we can guarantee that there exists δ 1 so that g′(x) 6 = 0 for all x in [r − δ 1 , r + δ 1 ] (note that this interval needs to be contained in (a, b), so make the choice of δ 1 appropriately). In this interval, F is continuous. Since we also assumed that g′′^ is continuous on (a, b), then F ′^ will also be continuous on [r − δ 1 , r + δ 1 ]
- (The size of F ′(x)) We know that F ′(r) = 0, and we know that F ′(x) is continuous on [r − δ 1 , r + δ 1 ]. By the continuity of F ′, there must exist 0 < δ 2 ≤ δ 1 so that |F ′(x)| < 1 for all x in [r − δ 2 , r + δ 2 ]. The value δ 2 is the δ in our Theorem.
By our two previous observations, we can apply the Fixed Point Convergence theorem to the function F - That proves both convergence and (at least) quadratic convergence.
0.2.1 EXERCISE:
Prove that, if g(r) = 0 and g is continuous on an interval about r, then there exists δ so that |g(x)| ≤ k < 1 for all x in (r − δ, r + δ). (HINT: Follow the proof of the Lemma)
0.2.2 EXERCISE:
Give an interval on which we can guarantee the convergence of Newton’s Method, if g(x) = x^2 − 1 (give the interval about the root r = 1).
0.2.3 Failure to Converge Quadratically:
Consider g(x) = x^2. Will Newton’s Method converge quadratically to the root x = 0? First, we convert to the Fixed Point Iteration method and see what we can conclude:
F (x) = x −
g(x) g′(x)
= x −
x^2 2 x
= x −
x 2
x
In this case, F ′(0) = 12 6 = 0, so Newton’s Method will converge, but only linearly. This happened because there was a multiple root at x = 0: Definition: A function f (x) has a root of multiplicity m at x = a if:
f (a) = 0, f ′(a) = 0, f ′′(a) = 0,... , f (m)(a) = 0, but f (m+1)(a) 6 = 0
Be sure to keep the conclusions of the Fixed Point Method and Newton’s Method distinct:
- In Fixed Point Iteration, if F ′(r) = 0, we get at least quadratic convergence. If F ′(r) 6 = 0, we get linear convergence.
- In Newton’s Method, if g′(r) 6 = 0, we get quadratic convergence, and if g′(r) = 0, we get only linear convergence.
0.2.4 Exercise:
The function f (x) = x^3 − 4 x has a root at r = 2. If the error ei = xi − r after 4 steps of Newton’s Method is e 4 = 10−^6 , estimate e 5. (Hint: What is the limit of en+1/en as n → ∞?)
0.2.5 Practical Notes for Newton’s Method:
If you believe your function has a “simple” root (that is, g(r) = 0 and g′(r) 6 = 0), and you have a reasonable starting guess, x 0 , then Newton’s Method is the method of choice. In practice, we will not know if we have a simple root, and we may not have a very good starting estimate- In that case, you might use something like Bisection to get you started, then switch to Newton’s Method once you’re close. We’ll see this in a method later on.
1 Methods without Derivatives
There are methods other than bisection if the function’s derivative is not known (or not easily computed). We discuss the ideas briefly below:
- Secant Method Rather than using the x−intercept of the tangent line (as in Newton), we begin with two points, x 0 and x 1 , then use the x−intercept of the secant line as our new estimate, x 2. In this method, we do not need to start with the root bracketed (as in bisection), and in fact the root may not be bracketed.
- Method of False Position Similar to bisection, but the midpoint is replaced by the x−intercept of the line between (a, f (a)) and (b, f (b)). The new interval is chosen so that the root is still bracketed. Note that this method may not do as well as bisection, which halves the interval at each step.
- How is the function being defined? It can be entered as an inline function (like our examples), or as the name of an M-file.
- Did the user input an appropriate bracket for the root, or an initial guess, from which we have to construct an initial bracket? Is the initial guess appropriate (for example, did the user input −1 for
x)?
- How much output does the user want? The default is for Matlab to stay quiet until the final result, but you can have it output details about what it is doing. You might try this example:
fzero(’x.^3+x-1’,1,optimset(’Display’,’iter’))
You can see fzero trying to do an initial bracketing before the algorithm proceeds.
- How should the algorithm stop, and what information should be presented? Lots of options here.
