Gilbert Strang- Linear Algebra, Study notes of Linear Algebra

Download Gilbert Strang- Linear Algebra and more Study notes Linear Algebra in PDF only on Docsity!

Linear Algebra and Its Applications

Fourth Edition

Gilbert Strang

y

x  y z

z

Ax  b

b

Ay  b

Az  0

• 1 Matrices and Gaussian Elimination Preface iv
  • 1.1 Introduction
  • 1.2 The Geometry of Linear Equations
  • 1.3 An Example of Gaussian Elimination
  • 1.4 Matrix Notation and Matrix Multiplication
  • 1.5 Triangular Factors and Row Exchanges
  • 1.6 Inverses and Transposes
  • 1.7 Special Matrices and Applications
    • Review Exercises
• 2 Vector Spaces
  • 2.1 Vector Spaces and Subspaces
  • 2.2 Solving Ax = 0 and Ax = b
  • 2.3 Linear Independence, Basis, and Dimension
  • 2.4 The Four Fundamental Subspaces
  • 2.5 Graphs and Networks
  • 2.6 Linear Transformations
    • Review Exercises
• 3 Orthogonality
  • 3.1 Orthogonal Vectors and Subspaces
  • 3.2 Cosines and Projections onto Lines
  • 3.3 Projections and Least Squares
  • 3.4 Orthogonal Bases and Gram-Schmidt
  • 3.5 The Fast Fourier Transform
    • Review Exercises
• 4 Determinants ii CONTENTS - 4.1 Introduction - 4.2 Properties of the Determinant - 4.3 Formulas for the Determinant - 4.4 Applications of Determinants - Review Exercises
• 5 Eigenvalues and Eigenvectors - 5.1 Introduction - 5.2 Diagonalization of a Matrix - 5.3 Difference Equations and Powers Ak - 5.4 Differential Equations and eAt - 5.5 Complex Matrices - 5.6 Similarity Transformations - Review Exercises
  • 6 Positive Definite Matrices
    • 6.1 Minima, Maxima, and Saddle Points
    • 6.2 Tests for Positive Definiteness
    • 6.3 Singular Value Decomposition
    • 6.4 Minimum Principles
    • 6.5 The Finite Element Method
  • 7 Computations with Matrices
    • 7.1 Introduction
    • 7.2 Matrix Norm and Condition Number
    • 7.3 Computation of Eigenvalues
    • 7.4 Iterative Methods for Ax = b
  • 8 Linear Programming and Game Theory
    • 8.1 Linear Inequalities
    • 8.2 The Simplex Method
    • 8.3 The Dual Problem
    • 8.4 Network Models
    • 8.5 Game Theory
  • A Intersection, Sum, and Product of Spaces
    • A.1 The Intersection of Two Vector Spaces
    • A.2 The Sum of Two Vector Spaces
    • A.3 The Cartesian Product of Two Vector Spaces
    • A.4 The Tensor Product of Two Vector Spaces
    • A.5 The Kronecker Product A ⊗ B of Two Matrices

CONTENTS iii

B The Jordan Form 466

C Matrix Factorizations 473

D Glossary: A Dictionary for Linear Algebra 475

E MATLAB Teaching Codes 484

F Linear Algebra in a Nutshell 486

AT~y = ~ 0

A~x = ~ 0

~ 0 ~^0

Rn^ Rm

Row Space Column Space all AT~y all A~x

Null Space (^) Left Null Space

A~x = ~b

AT~y = ~c

C(AT) dim r

C(A) dim r

N (A) dim n − r

N (AT) dim m − r

v

Matrices

I will keep going a little more to convert combinations of three-dimensional vectors into linear algebra. If the vectors are v = ( 1 , 2 , 3 ) and w = ( 1 , 3 , 4 ), put them into the columns of a matrix:

matrix =

To find combinations of those columns, “multiply” the matrix by a vector (c, d):

Linear combinations cv + dw

[

c d

]

= c

 +^ d

Those combinations fill a vector space. We call it the column space of the matrix. (For these two columns, that space is a plane.) To decide if b = ( 2 , 5 , 7 ) is on that plane, we have three components to get right. So we have three equations to solve:  

[

c d

]

 means

c + d = 2 2 c + 3 d = 5 3 c + 4 d = 7

I leave the solution to you. The vector b = ( 2 , 5 , 7 ) does lie in the plane of v and w. If the 7 changes to any other number, then b won’t lie in the plane—it will not be a combination of v and w, and the three equations will have no solution. Now I can describe the first part of the book, about linear equations Ax = b. The matrix A has n columns and m rows. Linear algebra moves steadily to n vectors in m- dimensional space. We still want combinations of the columns (in the column space). We still get m equations to produce b (one for each row). Those equations may or may not have a solution. They always have a least-squares solution.

The interplay of columns and rows is the heart of linear algebra. It’s not totally easy, but it’s not too hard. Here are four of the central ideas:

1. The column space (all combinations of the columns).
2. The row space (all combinations of the rows).
3. The rank (the number of independent columns) (or rows).
4. Elimination (the good way to find the rank of a matrix).

I will stop here, so you can start the course.

vi PREFACE

Web Pages

It may be helpful to mention the web pages connected to this book. So many messages come back with suggestions and encouragement, and I hope you will make free use of everything. You can directly access http://web.mit.edu/18.06, which is continually updated for the course that is taught every semester. Linear algebra is also on MIT’s OpenCourseWare site http://ocw.mit.edu, where 18.06 became exceptional by including videos of the lectures (which you definitely don’t have to watch...). Here is a part of what is available on the web:

1. Lecture schedule and current homeworks and exams with solutions.
2. The goals of the course, and conceptual questions.
3. Interactive Java demos (audio is now included for eigenvalues).
4. Linear Algebra Teaching Codes and MATLAB problems.
5. Videos of the complete course (taught in a real classroom). The course page has become a valuable link to the class, and a resource for the students. I am very optimistic about the potential for graphics with sound. The bandwidth for voiceover is low, and FlashPlayer is freely available. This offers a quick review (with active experiment), and the full lectures can be downloaded. I hope professors and students worldwide will find these web pages helpful. My goal is to make this book as useful as possible with all the course material I can provide.

Other Supporting Materials

Student Solutions Manual 0-495-01325-0 The Student Solutions Manual provides solutions to the odd-numbered problems in the text.

Instructor’s Solutions Manual 0-030-10588-4 The Instructor’s Solutions Man- ual has teaching notes for each chapter and solutions to all of the problems in the text.

Structure of the Course

The two fundamental problems are Ax = b and Ax = λ x for square matrices A. The first problem Ax = b has a solution when A has independent columns. The second problem Ax = λ x looks for independent eigenvectors. A crucial part of this course is to learn what “independence” means. I believe that most of us learn first from examples. You can see that

A =

 does not have independent columns.

viii PREFACE

Acknowledgments

I enjoyed writing this book, and I certainly hope you enjoy reading it. A big part of the pleasure comes from working with friends. I had wonderful help from Brett Coonley and Cordula Robinson and Erin Maneri. They created the LATEX files and drew all the figures. Without Brett’s steady support I would never have completed this new edition. Earlier help with the Teaching Codes came from Steven Lee and Cleve Moler. Those follow the steps described in the book; MATLAB and Maple and Mathematica are faster for large matrices. All can be used (optionally) in this course. I could have added “Factorization” to that list above, as a fifth avenue to the understanding of matrices:

[L, U, P] = lu(A) for linear equations [Q, R] = qr(A) to make the columns orthogonal [S, E] = eig(A) to find eigenvectors and eigenvalues. In giving thanks, I never forget the first dedication of this textbook, years ago. That was a special chance to thank my parents for so many unselfish gifts. Their example is an inspiration for my life. And I thank the reader too, hoping you like this book.

Gilbert Strang

Chapter 1

Matrices and Gaussian Elimination

1.1 Introduction

This book begins with the central problem of linear algebra: solving linear equations. The most important ease, and the simplest, is when the number of unknowns equals the number of equations. We have n equations in n unknowns, starting with n = 2:

Two equations 1 x + 2 y = 3 Two unknowns 4 x + 5 y = 6.

The unknowns are x and y. I want to describe two ways, elimination and determinants, to solve these equations. Certainly x and y are determined by the numbers 1, 2, 3, 4, 5,

1. The question is how to use those six numbers to solve the system.
   1. Elimination Subtract 4 times the first equation from the second equation. This eliminates x from the second equation. and it leaves one equation for y:

(equation 2) − 4 (equation 1) − 3 y = − 6. (2)

Immediately we know y = 2. Then x comes from the first equation 1x + 2 y = 3:

Back-substitution 1 x + 2 ( 2 ) = 3 gives x = − 1. (3)

Proceeding carefully, we cheek that x and y also solve the second equation. This should work and it does: 4 times (x = −1) plus 5 times (y = 2) equals 6.

1. Determinants The solution y = 2 depends completely on those six numbers in the equations. There most be a formula for y (and also x) It is a “ratio of determinants” and I hope you will allow me to write it down directly:

y =

1.1 Introduction 3

y

x

b x (^) y= = 2 − (^1) x + 2y = 3

4 x + 5y = 6

One solution (x, y) = (− 1 , 2)

y

x

x + 2y = 3

4 x + 8y = 6

Parallel: No solution

y

x

x + 2y = 3

4 x + 8y = 12

Whole line of solutions Figure 1.1: The example has one solution. Singular cases have none or too many.

First we have to introduce matrices and vectors and the rules for multiplication. Every matrix has a transpose AT. This matrix has an inverse A−^1.

1. In most cases elimination goes forward without difficulties. The matrix has an inverse and the system Ax = b has one solution. In exceptional cases the method will break down—either the equations were written in the wrong order, which is easily fixed by exchanging them, or the equations don’t have a unique solution. That singular case will appear if 8 replaces 5 in our example:

Singular case Two parallel lines

1 x + 2 y = 3 4 x + 8 y = 6.

Elimination still innocently subtracts 4 times the first equation from the second. But look at the result!

(equation 2) − 4 (equation 1) 0 = − 6.

This singular case has no solution. Other singular cases have infinitely many solu- tions. (Change 6 to 12 in the example, and elimination will lead to 0 = 0. Now y can have any value,) When elimination breaks down, we want to find every possible solution.

1. We need a rough count of the number of elimination steps required to solve a sys- tem of size n. The computing cost often determines the accuracy in the model. A hundred equations require a third of a million steps (multiplications and subtrac- tions). The computer can do those quickly, but not many trillions. And already after a million steps, roundoff error could be significant. (Some problems are sen- sitive; others are not.) Without trying for full detail, we want to see large systems that arise in practice, and how they are actually solved.

The final result of this chapter will be an elimination algorithm that is about as effi- cient as possible. It is essentially the algorithm that is in constant use in a tremendous variety of applications. And at the same time, understanding it in terms of matrices—the coefficient matrix A, the matrices E for elimination and P for row exchanges, and the

4 Chapter 1 Matrices and Gaussian Elimination

final factors L and U—is an essential foundation for the theory. I hope you will enjoy this book and this course.

1.2 The Geometry of Linear Equations

The way to understand this subject is by example. We begin with two extremely humble equations, recognizing that you could solve them without a course in linear algebra. Nevertheless I hope you will give Gauss a chance:

2 x − y = 1 x + y = 5.

We can look at that system by rows or by columns. We want to see them both. The first approach concentrates on the separate equations (the rows). That is the most familiar, and in two dimensions we can do it quickly. The equation 2x − y = 1 is represented by a straight line in the x-y plane. The line goes through the points x = 1, y = 1 and x = 12 , y = 0 (and also through ( 2 , 3 ) and all intermediate points). The second equation x + y = 5 produces a second line (Figure 1.2a). Its slope is dy/dx = −1 and it crosses the first line at the solution. The point of intersection lies on both lines. It is the only solution to both equations. That point x = 2 and y = 3 will soon be found by “elimination.”

b

b

(0, 5)

(0, −1) (

1 2 ,^ 0)^ x + y = 5

2 x − y = 1

x

y

(5, 0)

(x, y) = (2, 3)

(a) Lines meet at x = 2, y = 3

b b

b b

b (− 3 , 3)

(− 1 , 1) (2, 1) = column 1

(4, 2)

(1, 5) = 2 (column 1) +3 (column 2)

(b) Columns combine with 2 and 3

Figure 1.2: Row picture (two lines) and column picture (combine columns).

The second approach looks at the columns of the linear system. The two separate equations are really one vector equation:

Column form x

[

]

• y

[

]

[

]

6 Chapter 1 Matrices and Gaussian Elimination

The second plane is 4u − 6 v = −2. It is drawn vertically, because w can take any value. The coefficient of w is zero, but this remains a plane in 3-space. (The equation 4 u = 3, or even the extreme case u = 0, would still describe a plane.) The figure shows the intersection of the second plane with the first. That intersection is a line. In three dimensions a line requires two equations; in n dimensions it will require n − 1. Finally the third plane intersects this line in a point. The plane (not drawn) represents the third equation − 2 u + 7 v + 2 w = 9, and it crosses the line at u = 1, v = 1, w = 2. That triple intersection point ( 1 , 1 , 2 ) solves the linear system. How does this row picture extend into n dimensions? The n equations will con- tain n unknowns. The first equation still determines a “plane.” It is no longer a two- dimensional plane in 3-space; somehow it has “dimension” n − 1. It must be flat and extremely thin within n-dimensional space, although it would look solid to us. If time is the fourth dimension, then the plane t = 0 cuts through four-dimensional space and produces the three-dimensional universe we live in (or rather, the universe as it was at t = 0). Another plane is z = 0, which is also three-dimensional; it is the ordinary x-y plane taken over all time. Those three-dimensional planes will intersect! They share the ordinary x-y plane at t = 0. We are down to two dimensions, and the next plane leaves a line. Finally a fourth plane leaves a single point. It is the intersection point of 4 planes in 4 dimensions, and it solves the 4 underlying equations. I will be in trouble if that example from relativity goes any further. The point is that linear algebra can operate with any number of equations. The first equation produces an (n − 1 )-dimensional plane in n dimensions, The second plane intersects it (we hope) in a smaller set of “dimension n − 2.” Assuming all goes well, every new plane (every new equation) reduces the dimension by one. At the end, when all n planes are accounted for, the intersection has dimension zero. It is a point, it lies on all the planes, and its coordinates satisfy all n equations. It is the solution!

Column Vectors and Linear Combinations

We turn to the columns. This time the vector equation (the same equation as (1)) is

Column form u

 + v

 + w

 = b. (2)

Those are three-dimensional column vectors. The vector b is identified with the point whose coordinates are 5 , − 2 , 9. Every point in three-dimensional space is matched to a vector, and vice versa. That was the idea of Descartes, who turned geometry into algebra by working with the coordinates of the point. We can write the vector in a column, or we can list its components as b = ( 5 , − 2 , 9 ), or we can represent it geometrically by an arrow from the origin. You can choose the arrow, or the point, or the three numbers. In six dimensions it is probably easiest to choose the six numbers.

1.2 The Geometry of Linear Equations 7

We use parentheses and commas when the components are listed horizontally, and square brackets (with no commas) when a column vector is printed vertically. What really matters is addition of vectors and multiplication by a scalar (a number). In Figure 1.4a you see a vector addition, component by component:

Vector addition

In the right-hand figure there is a multiplication by 2 (and if it had been −2 the vector

b

b

b

b

[ (^0) − 02

]

[ (^5) (^00)

]

b =

[ (^5) − 2 9

]

[ (^0) (^09)

]

(a) Add vectors along axes

b

b b

[ (^2) (^04)

] = 2

[ (^1) (^02)

]

2 (column 3) [ (^2) −^42

]

[ (^1) − 76

]

[ (^3) − 52

]

columns 1 + 2

[ (^5) − 91

] = linear combination equals b

(b) Add columns 1 + 2 + (3 + 3)

Figure 1.4: The column picture: linear combination of columns equals b.

would have gone in the reverse direction):

Multiplication by scalars 2

Also in the right-hand figure is one of the central ideas of linear algebra. It uses both of the basic operations; vectors are multiplied by numbers and then added. The result is called a linear combination, and this combination solves our equation:

Linear combination 1

Equation (2) asked for multipliers u, v, w that produce the right side b. Those numbers are u = 1 , v = 1 , w = 2. They give the correct combination of the columns. They also gave the point ( 1 , 1 , 2 ) in the row picture (where the three planes intersect).

1.2 The Geometry of Linear Equations 9

Another singular system, close to this one, has an infinity of solutions. When the 6 in the last equation becomes 7, the three equations combine to give 0 = 0. Now the third equation is the sum of the first two. In that case the three planes have a whole line in common (Figure 1.5c). Changing the right sides will move the planes in Figure 1.5b parallel to themselves, and for b = ( 2 , 5 , 7 ) the figure is suddenly different. The lowest plane moved up to meet the others, and there is a line of solutions. Problem 1.5c is still singular, but now it suffers from too many solutions instead of too few. The extreme case is three parallel planes. For most right sides there is no solution (Figure 1.5d). For special right sides (like b = ( 0 , 0 , 0 )!) there is a whole plane of solutions—because the three parallel planes move over to become the same. What happens to the column picture when the system is singular? it has to go wrong; the question is how, There are still three columns on the left side of the equations, and we try to combine them to produce b. Stay with equation (3):

Singular case: Column picture Three columns in the same plane Solvable only for b in that plane

u

 +^ v

 +^ w

 =^ b.^ (4)

For b = ( 2 , 5 , 7 ) this was possible; for b = ( 2 , 5 , 6 ) it was not. The reason is that those three columns lie in a plane. Then every combination is also in the plane (which goes through the origin). If the vector b is not in that plane, no solution is possible (Figure 1.6). That is by far the most likely event; a singular system generally has no solution. But there is a chance that b does lie in the plane of the columns. In that case there are too many solutions; the three columns can be combined in infinitely many ways to produce b. That column picture in Figure 1.6b corresponds to the row picture in Figure 1.5c.

b

b

b not in place

3 columns in a plane

(a) no solution

b

b in place b

3 columns in a plane

(b) infinity of solutions

Figure 1.6: Singular cases: b outside or inside the plane with all three columns.

How do we know that the three columns lie in the same plane? One answer is to find a combination of the columns that adds to zero. After some calculation, it is u = 3, v = 1, w = −2. Three times column 1 equals column 2 plus twice column 3. Column 1 is in

10 Chapter 1 Matrices and Gaussian Elimination

the plane of columns 2 and 3. Only two columns are independent. The vector b = ( 2 , 5 , 7 ) is in that plane of the columns—it is column 1 plus column 3—so (1, 0, 1) is a solution. We can add an multiple of the combination ( 3 , − 1 , − 2 ) that gives b = 0. So there is a whole line of solutions—as we know from the row picture. The truth is that we knew the columns would combine to give zero, because the rows did. That is a fact of mathematics, not of computation—and it remains true in dimension n. If the n planes have no point in common, or infinitely many points, then the n columns lie in the same plane. If the row picture breaks down, so does the column picture. That brings out the difference between Chapter 1 and Chapter 2. This chapter studies the most important problem—the nonsingular case—where there is one solution and it has to be found. Chapter 2 studies the general case, where there may be many solutions or none. In both cases we cannot continue without a decent notation (matrix notation) and a decent algorithm (elimination). After the exercises, we start with elimination.

Problem Set 1.

1. For the equations x + y = 4, 2x − 2 y = 4, draw the row picture (two intersecting lines) and the column picture (combination of two columns equal to the column vector ( 4 , 4 ) on the right side).
2. Solve to find a combination of the columns that equals b:

Triangular system

u − v − w = b 1 v + w = b 2 w = b 3.

1. (Recommended) Describe the intersection of the three planes u + v + w + z = 6 and u + w + z = 4 and u + w = 2 (all in four-dimensional space). Is it a line or a point or an empty set? What is the intersection if the fourth plane u = −1 is included? Find a fourth equation that leaves us with no solution.
2. Sketch these three lines and decide if the equations are solvable:

3 by 2 system

x + 2 y = 2 x − y = 2 y = 1. What happens if all right-hand sides are zero? Is there any nonzero choice of right- hand sides that allows the three lines to intersect at the same point?

1. Find two points on the line of intersection of the three planes t = 0 and z = 0 and x + y + z + t = 1 in four-dimensional space.