Download Limits and Derivatives: Understanding Behavior of Functions and more Study notes Mathematical Methods for Numerical Analysis and Optimization in PDF only on Docsity!
Jim Lambers Math 105A Summer Session I 2003- Lecture 1 Notes
These notes correspond to Section 1.1 in the text.
What is Numerical Analysis?
In scientific applications, there are many situations in which mathematics is employed to answer questions that we may have about certain phenomena that we have observed. In such situations, a mathematical model is developed in order to describe this phenomena, and from this model, we can obtain a mathematical problem whose solution will, hopefully, help provide the answers that we seek. In almost all cases, these mathematical problems cannot be solved exactly. Therefore, it is necessary to design an algorithm that can yield an approximate solution, and then implement this algorithm in order to compute this approximate solution. Because the computed solution is only an approximation, it is also necessary to analyze our algorithm in order to determine whether the approximation is sufficiently accurate. If it can be determined that the computed solution is sufficiently accurate, then it can be interpreted in order to provide answers to our original questions about the phenomena that we are modeling. The process of designing, implementing and analyzing an algorithm for computing an approxi- mate solution to a mathematical problem arising from a scientific application is known as scientific computing. In a broader context, the discipline of computing approximate solutions of mathematical problems, regardless of the applications from which they arise, is known as numerical analysis.
Review of Calculus
Among the mathematical problems that can be solved using techniques from numerical analysis are the basic problems of differential and integral calculus:
- computing the instantaneous rate of change of one quantity with respect to another, which is a derivative, and
- computing the total change in a function over some portion of its domain, which is a definite integral.
Calculus also plays an essential role in the development and analysis of techniques used in numerical analysis, including those techniques that are applied to problems not arising directly from calculus. Therefore, it is appropriate to review some basic concepts from calculus before we begin our study of numerical analysis.
Limits
The basic problems of differential and integral calculus described in the previous paragraph can be solved by computing a sequence of approximations to the desired quantity and then determining what value, if any, the sequence of approximations approaches. This value is called a limit of the sequence. As a sequence is a function, we begin by defining, precisely, the concept of the limit of a function.
Definition We write lim x→a f (x) = L
if for any open interval I 1 containing L, there is some open interval I 2 containing a such that f (x) ∈ I 1 whenever x ∈ I 2 , and x 6 = a. We say that L is the limit of f (x) as x approaches a. We write lim x→a−^
f (x) = L
if, for any open interval I 1 containing L, there is an open interval I 2 of the form (c, a), where c < a, such that f (x) ∈ I 1 whenever x ∈ I 2. We say that L is the limit of f (x) as x approaches a from the left, or the left-hand limit of f (x) as x approaches a. Similarly, we write lim x→a+^
f (x) = L
if, for any open interval I 1 containing L, there is an open interval I 2 of the form (a, c), where c > a, such that f (x) ∈ I 1 whenever x ∈ I 2. We say that L is the limit of f (x) as x approaches a from the right, or the right-hand limit of f (x) as x approaches a.
We can make the definition of a limit a little more concrete by imposing sizes on the intervals I 1 and I 2 , as long as the interval I 1 can still be of arbitrary size. It can be shown that the following definition is equivalent to the previous one.
Definition We write lim x→a f (x) = L
if, for any > 0 , there exists a number δ > 0 such that |f (x) − L| < whenever |x − a| < δ, and x 6 = a.
Similar definitions can be given for the left-hand and right-hand limits. Note that in either definition, the point x = a is specifically excluded from consideration when requiring that f (x) be close to L whenever x is close to a. This is because the concept of a limit is only intended to describe the behavior of f (x) near x = a, as opposed to its behavior at x = a. Later in this lecture we discuss the case where the two distinct behaviors coincide.
- [a, b) if f is continuous on (a, b), and continuous from the right at a.
- (a, b] if f is continuous on (a, b), and continuous from the left at b.
- [a, b] if f is continuous on (a, b), continuous from the right at a, and continuous from the left at b.
In numerical analysis, it is often necessary to construct a continuous function, such as a polyno- mial, based on data obtained by measurements and problem-dependent constraints. In this course, we will learn some of the most basic techniques for constructing such continuous functions by a process called interpolation.
The Intermediate Value Theorem
Suppose that a function f is continuous on some closed interval [a, b]. The graph of such a function is a continuous curve connecting the points (a, f (a)) with (b, f (b)). If one were to draw such a graph, their pen would not leave the paper in the process, and therefore it would be impossible to “avoid” any y-value between f (a) and f (b). This leads to the following statement about such continuous functions.
Theorem (Intermediate Value Theorem) Let f be continuous on [a, b]. Then, on (a, b), f assumes every value between f (a) and f (b); that is, for any value y between f (a) and f (b), f (c) = y for some c in (a, b).
The Intermediate Value Theorem has a very important application in the problem of finding solutions of a general equation of the form f (x) = 0, where x is the solution we wish to compute and f is a given continuous function. Often, methods for solving such an equation try to identify an interval [a, b] where f (a) > 0 and f (b) < 0, or vice versa. In either case, the Intermediate Value Theorem states that f must assume every value between f (a) and f (b), and since 0 is one such value, it follows that the equation f (x) = 0 must have a solution somewhere in the interval (a, b). We can find an approximation to this solution using a procedure called bisection, which re- peatedly applies the Intermediate Value Theorem to smaller and smaller intervals that contain the solution. We will study bisection, and other methods for solving the equation f (x) = 0, in this course.
Derivatives
The basic problem of differential calculus is computing the instantaneous rate of change of one quantity y with respect to another quantity x. For example, y may represent the position of an object and x may represent time, in which case the instantaneous rate of change of y with respect to x is interpreted as the velocity of the object.
When the two quantities x and y are related by an equation of the form y = f (x), it is certainly convenient to describe the rate of change of y with respect to x in terms of the function f. Because the instantaneous rate of change is so commonplace, it is practical to assign a concise name and notation to it, which we do now.
Definition (Derivative) The derivative of a function f (x) at x = a, denoted by f ′(a), is
f ′(a) = lim h→ 0
f (a + h) − f (a) h
provided that the above limit exists. When this limit exists, we say that f is differentiable at a.
Remark Given a function f (x) that is differentiable at x = a, the following numbers are all equal:
- the derivative of f at x = a, f ′(a),
- the slope of the tangent line of f at the point (a, f (a)), and
- the instantaneous rate of change of y = f (x) with respect to x at x = a.
This can be seen from the fact that all three numbers are defined in the same way. 2
Many functions can be differentiated using differentiation rules such as those learned in a cal- culus course. However, many functions cannot be differentiated using these rules. For example, we may need to compute the instantaneous rate of change of a quantity y = f (x) with respect to another quantity x, where our only knowledge of the function f that relates x and y is a set of pairs of x-values and y-values that may be obtained using measurements. In this course we will learn how to approximate the derivative of such a function using this limited information. The most common methods involve constructing a continuous function, such as a polynomial, based on the given data, using interpolation. The polynomial can then be differentiated using differentiation rules. Since the polynomial is an approximation to the function f (x), its derivative is an approximation to f ′(x).
Differentiability and Continuity
Consider a tangent line of a function f at a point (a, f (a)). When we consider that this tangent line is the limit of secant lines that can cross the graph of f at points on either side of a, it seems impossible that f can fail to be continuous at a. The following result confirms this: a function that is differentiable at a given point (and therefore has a tangent line at that point) must also be continuous at that point.
Theorem If f is differentiable at a, then f is continuous at a.
It is important to keep in mind, however, that the converse of the above statement, “if f is continuous, then f is differentiable”, is not true. It is actually very easy to find examples of functions that are continuous at a point, but fail to be differentiable at that point. As an extreme
widths of the subintervals, Rn converges to the net area between the graph of f and the x-axis, where area below the x-axis is counted negatively. We define the definite integral of f (x) from a to b by ∫ (^) b
a
f (x) dx = lim n→∞ Rn,
where the sequence of Riemann sums {Rn}∞ n=1 is defined so that δ(n) → 0 as n → ∞, as in the previous discussion. The function f (x) is called the integrand, and the values a and b are the lower and upper limits of integration, respectively. The process of computing an integral is called integration. In this course, we will study the problem of computing an approximation to the definite integral of a given function f (x) over an interval [a, b]. We will learn a number of techniques for computing such an approximation, and all of these techniques involve the computation of an appropriate Riemann sum.
Extreme Values
In many applications, it is necessary to determine where a given function attains its minimum or maximum value. For example, a business wishes to maximize profit, so it can construct a function that relates its profit to variables such as payroll or maintenance costs. We now consider the basic problem of finding a maximum or minimum value of a general function f (x) that depends on a single independent variable x. First, we must precisely define what it means for a function to have a maximum or minimum value.
Definition (Absolute extrema) A function f has a absolute maximum or global maximum at c if f (c) ≥ f (x) for all x in the domain of f. The number f (c) is called the maximum value of f on its domain. Similarly, f has a absolute minimum or global minimum at c if f (c) ≤ f (x) for all x in the domain of f. The number f (c) is then called the minimum value of f on its domain. The maximum and minimum values of f are called the extreme values of f , and the absolute maximum and minimum are each called an extremum of f.
Before computing the maximum or minimum value of a function, it is natural to ask whether it is possible to determine in advance whether a function even has a maximum or minimum, so that effort is not wasted in trying to solve a problem that has no solution. The following result is very helpful in answering this question.
Theorem (Extreme Value Theorem) If f is continuous on [a, b], then f has an absolute maximum and an absolute minimum on [a, b].
Now that we can easily determine whether a function has a maximum or minimum on a closed interval [a, b], we can develop an method for actually finding them. It turns out that it is easier to find points at which f attains a maximum or minimum value in a “local” sense, rather than
a “global” sense. In other words, we can best find the absolute maximum or minimum of f by finding points at which f achieves a maximum or minimum with respect to “nearby” points, and then determine which of these points is the absolute maximum or minimum. The following definition makes this notion precise.
Definition (Local extrema) A function f has a local maximum at c if f (c) ≥ f (x) for all x in an open interval containing c. Similarly, f has a local minimum at c if f (c) ≤ f (x) for all x in an open interval containing c. A local maximum or local minimum is also called a local extremum.
At each point at which f has a local maximum, the function either has a horizontal tangent line, or no tangent line due to not being differentiable. It turns out that this is true in general, and a similar statement applies to local minima. To state the formal result, we first introduce the following definition, which will also be useful when describing a method for finding local extrema.
Definition(Critical Number) A number c in the domain of a function f is a critical number of f if f ′(c) = 0 or f ′(c) does not exist.
The following result describes the relationship between critical numbers and local extrema.
Theorem (Fermat’s Theorem) If f has a local minimum or local maximum at c, then c is a critical number of f ; that is, either f ′(c) = 0 or f ′(c) does not exist.
This theorem suggests that the maximum or minimum value of a function f (x) can be found by solving the equation f ′(x) = 0. As mentioned previously, we will be learning techniques for solving such equations in this course. These techniques play an essential role in the solution of problems in which one must compute the maximum or minimum value of a function, subject to constraints on its variables. Such problems are called optimization problems. Although we will not discuss optimization problems in this course, we will learn about some of the building blocks of methods for solving these very important problems.
The Mean Value Theorem
While the derivative describes the behavior of a function at a point, we often need to understand how the derivative influences a function’s behavior on an interval. This understanding is essential in numerical analysis because, it is often necessary to approximate a function f (x) by a function g(x) using knowledge of f (x) and its derivatives at various points. It is therefore natural to ask how well g(x) approximates f (x) away from these points. The following result, a consequence of Fermat’s Theorem, gives limited insight into the rela- tionship between the behavior of a function on an interval and the value of its derivative at a point.
Theorem (Rolle’s Theorem) If f is continuous on a closed interval [a, b] and is differentiable on the open interval (a, b), and if f (a) = f (b), then f ′(c) = 0 for some number c in (a, b).
In other words, f assumes its average value over [a, b], defined by
fave =
b − a
∫ (^) b
a
f (x) dx,
at some point in [a, b], just as the Mean Value Theorem states that the derivative of a function assumes its average value over an interval at some point in the interval. The Mean Value Theorem for Integrals is also a special case of the following more general result.
Theorem (Weighted Mean Value Theorem for Integrals) If f is continuous on [a, b], and g is a function that is integrable on [a, b] and does not change sign on [a, b], then
∫ (^) b
a
f (x)g(x) dx = f (c)
∫ (^) b
a
g(x) dx
for some c in (a, b).
In the case where g(x) is a function that is easy to antidifferentiate and f (x) is not, this theorem can be used to obtain an estimate of the integral of f (x)g(x) over an interval.
Taylor’s Theorem
In many cases, it is useful to approximate a given function f (x) by a polynomial, because one can work much more easily with polynomials than with other types of functions. As such, it is necessary to have some insight into the accuracy of such an approximation. The following theorem, which is a consequence of the Weighted Mean Value Theorem for Integrals, provides this insight.
Theorem (Taylor’s Theorem) Let f be n times continuously differentiable on an interval [a, b], and suppose that f (n+1)^ exists on [a, b]. Let x 0 ∈ [a, b]. Then, for any point x ∈ [a, b],
f (x) = Pn(x) + Rn(x),
where
Pn(x) =
∑^ n
j=
f (j)(x 0 ) j! (x − x 0 )j
= f (x 0 ) + f ′(x 0 )(x − x 0 ) +
f ′′(x 0 )(x − x 0 )^2 + · · · +
f (n)(x 0 ) n! (x − x 0 )n
and
Rn(x) =
f (n+1)(ξ(x)) (n + 1)! (x − x 0 )n+1,
where ξ(x) is between x 0 and x.
The polynomial Pn(x) is the nth Taylor polynomial of f with center x 0 , and the expression Rn(x) is called the Taylor remainder of Pn(x). As Pn(x) can be used to approximate f (x), the remainder Rn(x) is also referred to as the truncation error of Pn(x). When the center x 0 is zero, the nth Taylor polynomial is also known as the nth Maclaurin polynomial. Because approximation of functions by polynomials is employed in the development and analysis of many techniques in numerical analysis, the usefulness of Taylor’s Theorem cannot be overstated. In fact, it can be said that Taylor’s Theorem is the Fundamental Theorem of Numerical Analysis, just as the theorem describing inverse relationship between derivatives and integrals is called the Fundamental Theorem of Calculus.
