Notes Matrix and Linear Algebra, Schemes and Mind Maps of Linear Algebra

Download Notes Matrix and Linear Algebra and more Schemes and Mind Maps Linear Algebra in PDF only on Docsity!

Notes

Matrix and Linear Algebra

Prof. Xu Chen

Mechanical Engineering Building Rm 325

Department of Mechanical Engineering

University of Washington

chx AT uw.edu

Contents

1 Basic concepts of matrices and vectors

Matrices and vectors are the main tools of linear algebra. They provide great convenience in expressing

and manipulating large amounts of data and functions. Consider, for instance, a linear equation set

3 x 1 + 4x 2 + 10x 3 = 6

x 1 + 4x 2 − 10 x 3 = 5

4 x 2 + 10x 3 = − 1

This is equivalent to 

x 1

x 2

x 3

Formally, we write an m × n matrix A as

A = [ajk] =

a 11 a 12... a 1 n

a 21...... a 2 n

. .

.......

am 1 am 2... amn

m × n (reads m by n) is the dimension/size of the matrix. It means that A has m rows and n columns.

Each element ajk is an entry of the matrix. You can see that each entry is marked by two subscripts:

the first is the row number and the second is the column number. For two matrices A and B to be

equal, it must be that ajk = bjk for any j and k, i.e., all corresponding entries of the matrices must

equal. Thus, matrices of different sizes are always different.

If m = n, A belongs to the class of square matrices. The entries a 11 , a 22 ,... , ann are then called

the diagonal entries of A.

Upper triangular matrices are square matrices with nonzero entries only on and above the main

diagonal. Similarly, lower triangular matrices have nonzero entries only on and below the main diagonal.

Diagonal matrices have nonzero entries only on the main diagonal.

An identity matrix is a diagonal matrix whose nonzero elements are all 1.

Vectors are special matrices whose row or column number is one. A row vector has the form of

a = [a 1 , a 2 ,... , an]

Its dimension is 1 × n. An m × 1 column vector has the form of

b =

b 1

b 2

. . .

bm

Example (Matrix and quadratic forms). We can use matrices to express general quadratic functions

of vectors. For instance

f (x) = x

T Ax + 2bx + c

is equivalent to

f (x) =

[

x

1

]T [

A b

b

T c

] [

x

1

]

1.1 Matrix addition and multiplication

The sum of two matrices A and B (of the same size) is

A + B = [ajk + bjk]

The product between a m × n matrix A and a scalar c is

cA = [cajk]

i.e. each entry of A is multiplied by c to generate the corresponding entry of cA.

The matrix product C = AB is meaningful only if the column number of A equals the row number

of B. The computation is done as shown in the following example:

a 11 a 12 a 13

a 21 a 22 a 23

a 31 a 32 a 33

a 41 a 42 a 43

b 11 b 12

b 21 b 22

b 31 b 32

c 11 c 12

c 21 c 22

c 31 c 32

c 41 c 42

where

c 21 = a 21 b 11 + a 22 b 21 + a 23 b 31

= [a 21 , a 22 , a 23 ]

b 11

b 21

b 31

= "second row of A" × "first column of B"

More generally:

cjk = aj 1 b 1 k + aj 2 b 2 k + · · · + ajnbnk

= [aj 1 , aj 2 ,... , ajn]

b 1 k

b 2 k

. . .

bnk

namely, the jk entry of C is obtained by multiplying each entry in the jth row of A by the corresponding

entry in the kth column of B and then adding these n products. This is called a multiplication of rows

into columns.

Transposition has the following rules:

A

T

)T

= A

• (A + B)

T = A

T

• B

T

• (cA)

T = cA

T

• (AB)

T = B

T A

T

If A = A

T , then A is called symmetric. If A = −A

T then A is called skew-symmetric. We will talk

about these special matrices in more details later in this set of notes.

1.3 Exercises

1. (Lyapunov operator) Let

P =

P 11

P 12

P 13

P 12

P 22

P 23

P 13

P 23

P 33

 = P T

Show that

P

a 1

a 2

a 3

a

a

T (^03) × 1 03 × 1

(^03) × 1 a

T (^03) × 1

(^03) × 1 03 × 1 a

T

P 11

P 12

P 13

P 21 = P 12

P 22

P 23

P 31 = P 13

P 32 = P 23

P 33

a 1 a 2 a 3 0 0 0

0 a 1 0 a 2 a 3 0

0 0 a 1 0 a 2 a 3

P 11

P 12

P 13

P 22

P 23

P 33

Let A be an arbitrary three by three real matrix. Let Q = L (P ) = A

T P + P A and consider the

column expression of P, Q, A:

P 11

P 21

P 31

P∗ 1

P 12

P 22

P 23

P∗ 2

P 13

P 23

P 33

P∗ 3

Q 11

Q 21

Q 31

Q∗ 1

Q 12

Q 22

Q 23

Q∗ 2

Q 13

Q 23

Q 33

Q∗ 3

a 11

a 21

a 31 ︸ ︷︷ ︸ a∗ 1

a 12

a 22

a 23 ︸ ︷︷ ︸ a∗ 2

a 13

a 23

a 33 ︸ ︷︷ ︸ a∗ 3

Show that

Q∗ 1

Q∗ 2

Q∗ 3

A

T

A

T

A

T

P∗ 1

P∗ 2

P∗ 3

a

T ∗ 1 0 0

0 a

T ∗ 1 0

0 0 a

T ∗ 1

a

T ∗ 2 0 0

0 a

T ∗ 2 0

0 0 a

T ∗ 2

a

T ∗ 3 0 0

0 a

T ∗ 3 0

0 0 a

T ∗ 3

P∗ 1

P∗ 2

P∗ 3

2. (Convolution and filtering) Show that

e 0 e 1...... en

e 0

e 1

e 0

a 0

a 1

. . .

an− 1

a 0 a 1... an− 1 1

a 1.

..^

an− 1.

e 0

e 1

e 2

. . .

en

and

[e 0 ,... en− 1 , en]

a 0 a 1... an− 1 1

a 1.

..^

..^

an− 1.

= [a 0 ,... an− 1 , 1]

e 0

e 1 e 0

. . .

en...... e 1 e 0

Here, all unmarked entrices are zero. See, e.g., an application in ’Transmission of signal nons-

moothness and transient improvement in add-on servo control’, Tianyu Jiang and Xu Chen, IEEE

Transactions on Control Systems Technology, 26(2):486-496, 03, 2018.

3. (A linear equation set and its matrix form) Consider an n × n matrix X = [xij ] whose row sums

and column sums are all 1, i.e.,

n ∑

i=

xij = 1; ∀ j = 1, 2 ,... , n

∑^ n

j=

xij = 1; ∀ i = 1, 2 ,... , n

2 Linear systems of equations

A linear system of m equations in n unknowns x 1 ,... , xn is a set of equations of the form

a 11 x 1 + a 12 x 2 +... a 1 nxn = b 1

a 21 x 1 + a 22 x 2 +... a 2 nxn = b 2 (2)

am 1 x 1 + am 2 x 2 +... amnxn = bm

The system is linear because each variable xj appears in the first power only. a 11 ,... , amn are

the coefficients of the system. If all the bj are zero, then the linear equation is called a homogeneous

system. Otherwise, it is a nonhomogeneous system.

Homogeneous systems always have at least the trivial solution x 1 = x 2 = · · · = xn = 0.

The m equations (2) may be written as a single vector equation

Ax = b

where

A =

a 11 a 12...... a 1 n

a 21 a 22...... a 2 n

. . .

am 1 am 2...... amn

, x =

x 1

x 2

. . .

. . .

xn

, b =

b 1

b 2

. . .

bm

Consider the example of solving

x 1 + x 2 = 1

x 1 − x 2 = 20

Very quickly, you can obtain the solution of x 1 = 21/ 2 and x 2 = − 19 / 2. In a bit more details, here is

one solution procedure:

• Subtract the first equation from the second equation, yielding

− 2 x 2 = 19

and hence x 2 = − 19 / 2.

• Substitute x 2 = − 19 / 2 to the first equation, to get

x 1 = 1 − x 2 = 21/ 2

For larger systems, Gauss

1 elimination is a systematic method to solve linear equations. We demonstrate

the procedures via the following example. Let

Ax = b

where

A =

, b =

i.e.

x 1 − x 2 + x 3 = 0 (3)

−x 1 + x 2 − x 3 = 0 (4)

10 x 2 + 25x 3 = 90 (5)

20 x 1 + 10x 2 = 80 (6)

Gauss elimination is done as follows:

1. Obtain the augmented matrix of the system

[

A b

]

2. Perform elementary row operation on the augmented matrix, to obtain the Row Echelon Form.

The idea is to systematically manipulate coefficients for the variables such that individual equa-

tions become as simplified as possible. For instance, adding the first row to the second row

gives 

This is equivalent to doing the step of adding (3) to (4) to get

x 1 − x 2 + x 3 = 0 (7)

10 x 2 + 25x 3 = 90 (9)

20 x 1 + 10x 2 = 80 (10)

(^1) Johann Carl Friedrich Gauss, 1777-1855, German mathematician: contributed significantly to many fields, including

number theory, algebra, statistics, analysis, differential geometry, geodesy, geophysics, electrostatics, astronomy, Matrix

theory, and optics.

Gauss was an ardent perfectionist. He was never a prolific writer, refusing to publish work which he did not consider

complete and above criticism. Mathematical historian Eric Temple Bell estimated that, had Gauss published all of his

discoveries in a timely manner, he would have advanced mathematics by fifty years.

Elementary Row Operations for Matrices What we have done can be summarized by the following

elementary matrix row operations:

• Interchange of two rows
• Addition of a constant multiple of one row to another row
• Multiplication of a row by a nonzero constant c

Let the final row echelon form be denoted by

[

R f

]

We have

1. The two systems Ax = b and Rx = f are equivalent.
2. At the end of the Gauss elimination (before the back substitution), the row echelon form of the

augmented matrix will be

r 11 r 12......... r 1 n f 1

r 22......... r 2 n f 2

. .

. (^)......

rrr... rrn fr

fr+

. . .

fm

where all unfilled entries are zero.

3. The number of nonzero rows, r, in the row-reduced coefficient matrix R is called the rank of R

and also the rank of A.

4. Solution concepts:

(a) No solution. If r is less than m (meaning that R actually has at least one row of all 0 s) and

at least one of the numbers fr+1, fr+2,... , fm is not zero, then the system Rx = f is

inconsistent: No solution is possible. Therefore the system Ax = b is inconsistent as well.

(b) Unique solution. If the system is consistent and r = n, there is exactly one solution, which

can be found by back substitution.

(c) Infinitely many solutions exist if fr+1 = fr+2 =... = fm = 0. To obtain any of these

solutions, choose values of xr+1,... , xn arbitrarily. Then solve the r-th equation for xr

(in terms of those arbitrary values), then the (r − 1)-st equation for xr− 1 , and so on up the

line.

3 Vector space

3.1 Fields

Consider the set of real numbers R and the set of complex numbers C. Denote F as either R or C.

You can see that F has the following properties: ∀ w, z, u ∈ F

• w + z = z + w
• (w + z) + u = w + (z + u)
• (wz) u = w (zu)
• there exists elements 0 and 1 in F such that z + 0 = z and z · 1 = z
• ∀z ∈ F, ∃w ∈ F s.t. z + w = 0
• (inverse) ∀z ∈ F, z 6 = 0, ∃w ∈ F such that zw = 1.
• u (w + z) = uw + uz

Real and complex numbers are fundamental for science and engineering. They have various nice prop-

erties. The notion of fields generalizes these two important sets of numbers.

Definition 2 (Field). A field F is a set of elements called scalars together with two binary operations,

addition (+) and multiplication (·), such that take any α, β, γ ∈ F the following hold:

(a) closure: α · β ∈ F, α + β ∈ F

(b) commutativity: α · β = β · α, α + β = β + α

(c) associativity: α + (β + γ) = (α + β) + γ, α · (β · γ) = (α · β) · γ

(d) distribution: α · (β + γ) = α · β + α · γ

(e) identity:

∃ additive identity 0 ∈ F such that α + 0 = α

∃ multiplicative identity 1 ∈ F such that α · 1 = α

(f) inverse:

∀α ∈ F, ∃ an additive inverse −α ∈ F such that α + (−α) = 0

∀α ∈ F and α 6 = 0, ∃ a multiplicative inverse α

− 1 ∈ F such that α · α

− 1 = 1

There is no need for division or substraction in the definition above. The existance of inverse from

wz = 1 makes the notion of 1 /z = z

− 1 meaningful, which in turn makes division meaningful, namely, w z

actually means w

1 z

where 1 /z is the inverse of z.

Example 3. The following are fields

• R: the set of real numbers
• C: the set of complex numbers
• R (s): the set of rational functions in s with real coefficients, namely, if G ∈ R(s), G =

(b 0 + b 1 s + b 2 s

2

• · · · + bms

m )/(a 0 + a 1 s + a 2 s

2

• · · · + ans

n ).

Definition 4 (Vector space). A vector space (V, F) is a set of vectors V together with a field F

and two operations, vector-vector addition (+) and vector-scalar multiplication (◦) such that for any

α, β, γ ∈ F and any v, v 1 , v 2 , v 3 ∈ V the following hold:

(a) closure: v 1 + v 2 ∈ V, α ◦ v 1 ∈ V

(b) commutativity: v 1 + v 2 = v 2 + v 1

(c) associativity:

v 1 + (v 2 + v 3 ) = (v 1 + v 2 ) + v 3

α ◦ (β ◦ γ) = (α · β) ◦ γ

(d) distribution:

α ◦ (v 1 + v 2 ) = α ◦ v 1 + α ◦ v 2

(α + β) ◦ v 1 = α ◦ v 1 + β ◦ v 1

(e) identity:

∃ a zero vector 0 ∈ V such that v + 0 = v

∃ multiplicative identity 1 ∈ F such that 1 ◦ v = v

(f) additive inverse: ∃ −v ∈ V such that v + (−v) = 0

We shall simplify the multiplication notations and use · alone as the appropriate action will be clear

from context. We will also use just 0 for the both identities 0 ∈ F and 0 ∈ V.

Most of the times, the base field F is either R or C. We often simply use F without explicitly stating

the base field.

Example 5. (R, R) is a vector space (any field is a vector space itself); (R [s] , R) with formal addition

and scalar multiplication of polynomials is a vector space; (R [s] , C) is however not a vector space.

3.4 Subspaces

A subset U of V is called a subspace of V if U is also a vector space. For example,

{(x 1 , 0 , 0) : x 1 ∈ F}

is a subspace of F

3 .

To check whether U is a subspace of V we only need to check three things:

• additive identity: 0 ∈ U
• closed under addition: u, v ∈ U implies u + v ∈ U
• closed under scalar multiplication: a ∈ F and u ∈ U implies au ∈ U

These conditions insure that the results of normal operations in U “stay in U,” and hence forming a

sub vector space.

Example 6. The following is not a subspace

(x 1 , x 2 ) ∈ F

2 : x 2 = x 1 + 10

One benefit of introducing subspaces is the enabling of decompositions of vector spaces.

The sum of U 1 , ..., Um is the set of all possible sums of elements of U 1 , ..., Um. More specifically

U 1 + ... + Um = {u 1 + u 2 + · · · + um : u 1 ∈ U 1 , ..., um ∈ Um}

For instance, let

U =

(x, 0 , 0) ∈ F

3 : x ∈ F

W =

(0, y, 0) ∈ F

3 : y ∈ F

Then

U + W =

(x, y, 0) ∈ F

3 : x, y ∈ F

is also a subspace of F

3 .

We will be especially interested in cases where each vector in V can be uniquely represented by

u 1 + u 2 + · · · + um

where uj ∈ Uj and V = U 1 + U 2 + · · · + Um. In fact, this situation is so important that it has a

special name: direct sum, written V = U 1 ⊕ U 2 ⊕ · · · ⊕ Um. As an example, if

U =

(x, 0 , z) ∈ F

3 : x, z ∈ F

W =

(0, y, 0) ∈ F

3 : y ∈ F

then

F

3 = U ⊕ W

Direct sums of subspaces are analogous to disjoint unions of subsets. We have the following theorem.

Theorem 7. Suppose that U and W are subspaces of V. Then V = U⊕W if and only if V = U+W

and U ∩ W = { 0 }.

3.5 Finite-dimensional vector spaces

Given a set of m vectors a 1 , a 2 , ..., am with the same size,

k 1 a 1 + k 2 a 2 + · · · + kmam

is called a linear combination of the vectors. If

a 1 = k 2 a 2 + k 3 a 3 + · · · + kmam

then a 1 is said to be linearly dependent on a 2 , a 3 , ..., am. The set

{a 1 , a 2 ,... , am} (19)

Consider a set of n linearly independent vectors, a 1 , a 2 , ..., an, each with n components. All the

possible linear combinations of a 1 , a 2 , ..., an form the vector space R

n

. This is the span of the n

vectors.

Definition 9 (Basis). A basis of V is a set B of vectors in V, such that any v ∈ V can be uniquely

expressed as a finite linear combination of vectors in B.

Remark 10. Basis are not unique. For example, both 1 and − 1 are basis for R.

Theorem 11. Every finite-dimensional vector space has a basis

Theorem 12. Every linearly independent list of vectors in a finite-dimensional vector space can be

extended to a basis of the vector space

Theorem 13. Suppose V is finite dimensional and U is a subspace of V. Then there is a subspace

W of V such that V = U ⊕ W.

4 Matrix defines linear transformations between vector spaces

Now that we know vector spaces, we will develop some deeper understanding of matrices.

Example 14. A person X has two ID cards from two different companies. Suppose both companies

include personal information such as name, height, and birthday. The first company arranges the data

as:

x 1 = name

x 2 = height (in ft)

x 3 = birthday

and X’s ID is composed of

x 1 = X

x 2 = 6. 0

x 3 = 19901201

The second company arranges X’s information as

y 1 = X

y 2 = 6019901201

y 3 = 6 × 30 .48 = 182.88 (cm)

Namely,

y 1 = name

y 2 = 10

9 height (in ft) + birthday

y 3 = height (in cm)

The two different ID cards are related by

y 1

y 2

y 3

9 1

0 30. 48 0

A

x 1

x 2

x 3

So the same person has two seemingly different profiles in two companies. Matrix A above connects

the two profiles. If the first company wants to shift the data base of its entire employees, all that needs

to be done is perform a matrix-vector multiplication in (20).

More generally, matrices define linear transformations/mappings between vector spaces. A vector

can have different representations in two vector spaces, which however can be connected by some

corresponding transformation matrix.

Example. A vector

[

x 1

x 2

]

is rotated by an angle of θ in the 2-dimensional vector space. Let x 1 =

r cos α and x 2 = r sin α. The rotated vector has the following representation

y 1 = r cos (θ + α) = r cos θ cos α − r sin θ sin α

y 2 = r sin (θ + α) = r sin θ cos α + r cos θ sin α

namely, [ y 1

y 2

]

[

cos θ − sin θ

sin θ cos θ

] [

x 1

x 2

]

Let X and Y be any vector spaces. To each vector x ∈ X we assign a unique vector y ∈ Y. In

this way we have a mapping (or transformation) of X into Y. If we denote such a mapping by F ,

we can write F (x) = y. The vector y ∈ Y is called the image of x ∈ X under the mapping F.

L is called a linear transformation or linear mapping, if ∀v, x ∈ X and c ∈ R,

L (v + x) = L (v) + L (x)

L (cx) = cL (x)

The scalar c can be extended to a more general scalar in a field F. Suppose V and W are vector

spaces over the same field F, L is called a linear transformation on V to W, if for all α, β ∈ F and all

v, x ∈ V,

L (αv + βx) = αL (v) + βL (x)

Example 15 (Lyapunov operator). V = R

n×n , W = R

n×n

L (P ) = A

T P + P A

where P ∈ V, A ∈ R

n×n , defines a linear transformation.