Download MATH 242 Lecture 23: Regression & Constrained Optimization with Lagrange Multipliers and more Study notes Mathematics in PDF only on Docsity!
MATH 242, LECTURE 23
- Regression analysis
Definition 1. The vertical deviation of a function f (x) from some collection of data points {(xi, yi)} is the sum (y 1 − f (x 1 ))^2 + (y 2 − f (x 2 ))^2 + · · · + (yi − f (xi))^2 + · · ·.
Example 2. Find the line whose vertical deviation from the points (1, 1), (2, 3) and (3, 4) is minimal.
Some important features of this example:
- Though it looks like x and y should be our variables, the slope m and y-intercept b of the line are the “real” variables. Much as the coefficients of a quadratic polynomial are variables when we fit a parabola to data.
- Ultimately, the critical point is found as a solution of a system of linear equations.
Definition 3. The linear regression line for a collection of data is the linear function whose vertical deviation from that collection is minimal.
The ability to find such lines is programmed into statistical and data analysis software, as well as your calculators. The book gives explicit formulae which you may use for the homework, but for the exam it will be more important that you understand the way in which minimization techniques are employed. (This is another case where we are learning exactly what our calculators are doing behind the scenes).
Theorem 4. The linear regression line for the collection of data (x 1 , y 1 ), (x 2 , y 2 ),... , (xk, yk) is the function f (x) = mx + b such that the sum
(x 1 m + b − y 1 )^2 + · · · + (xkm + b − yk)^2
is minimized. Taking the partial derivatives and setting them to zero leads to a system of two linear equations in the variables m and b.
One of the main applications of linear regression is to fill in/ predict/ extrapolate values for a function from known values.
Example 5. Because of a computer error, some of the sales figures for a real estate company were lost. The sales figures (measured in millions of dollars) which are available for Gary Gladhand are: 1998 1999 2001 2003 0.9 1.5 1.9 2. Graph these, and then use the regression line to estimate/predict what he
sold in 2000 and what he will sell in 2004.
- Constrained optimization and Lagrange multipliers
2.1. Motivation: the need for additional tools in constrained optimization. In multivariable optimization, it is often the case that there is some equation which imposes relations among the variables under consideration. Such constraint equations arise naturally in at least two distinct ways:
- The equation represents a relation intrinsic to the problem, as for example when the variables represent money spent and there is one fixed limited source for the funds. We saw such problems at the end of last term. 1
2 MATH 242, LECTURE 23
- When optimizing a multivariable function over a region, one must check not only relative maxima and minima but values on the boundary of the region (as we did in linear programming). The boundary of a region is a curve defined by some equation, and we must focus our attention on that curve. Before continuing general discussion, we clarify what we mean by constraint equations by looking at examples which are manageable with techniques developed last term.
Example 6. Minimize the function f (x, y) = x^2 + y^2 + xy subject to the constraint y = − 3. Minimize it subject to the constraint x + y = 5.
What we see in these examples is that (in these cases) we can use a constraint equation to solve for one variable in terms of the other, substitute that expression into our function, and thus obtain a one-variable function to optimize. We were able to do such problems last term because we ultimately had a one-variable function to optimize. But what if I wanted to minimize the f (x, y) from the example subject to the constraint x^5 y^7 − 3 xy^2 = 2? I could not just solve for one variable in terms of the other, so a new method is needed.
2.2. Tangencies of level curves and the Lagrange multiplier equations. In order to understand the fundamental idea behind the Lagrange equations, we investigate simple examples, paying close attention to the level curves at and near the optimum point.
Example 7. Find the minimum, and graph the level curves and the constraint curve near that minimum for the function f (x, y) = x^2 + y^2 , constrained by x + y = 2 and y = x^2 − 2.
What we see is that at the optimum point, the level curve for the function and the constraint curve are tangent! This makes sense geometrically, as we can see with graphical illustrations. This observation leads to a way to find optimum points because of the following theorem (which we will not be able to justify).
Theorem 8. The slope of the level curve f (x, y) = c for the function f at any point (x, y) is given by m = − f fxy.
Therefore, the level curve of f (x, y) is tangent to the constraint curve g(x, y) = c when − f fxy = − g gxy or fx gx =^
fy gy.^ If we call the number that they are both equal to^ λ^ (pronounced “lamb-duh”), then we have that fx = λgx and fy = λgy. These are known as the first two Lagrange equations.
Theorem 9. The maximum and minimum values for the function f (x, y) subject to the constraint g(x, y) = k occur at points (x, y) for which the following three equations hold:
(1) fx(x, y) = λgx(x, y). (2) fy (x, y) = λgy (x, y). (3) g(x, y) = k. We first apply this theorem to see that it gives the same results we found in our previous, simple, examples.
