Download Notes on Linear Systems: Solving and Understanding Linear Equations and more Study notes Computer Science in PDF only on Docsity!
CS6964: Notes On Linear Systems
1 Linear Systems
Systems of equations that are linear in the unknowns are said to be linear systems For instance
ax 1 + bx 2 = c dx 1 + ex 2 = f
gives 2 equations and 2 unknowns. More generally we have
a 11 x 1 +... + a 1 N xN = b 1 a 21 x 1 +... + a 2 N xN = b 2
...... aM 1 x 1 +... + aM N xN = bM
This can be written in matrix representation as
Ax = b (1)
where (^)
a 11 a 12... a 1 N a 21 a 22... a 2 N
... aM 1 aM 2... aM N
and
b =
b 1 b 2 .. . bM
The solution is given by multiplying boths sides by A−^1.
A−^1 Ax = Ix = x = A−^1 b (4)
This is the same answer you would get if you isolated one variable at a time and then substituted back into the other equation. We will be concerned with solving linear equations with uknowns x under the conditions
- When M < N or M = N and the equations are degenerate or singular. Degeneracy happens when the equations are not linearly independent. The determi- nant of a square matrix is zero if and only if it’s singular. The number of linearly independent equations is called the rank of A. If the rank is less than the size, the matrix is said to be not of full rank. Because of this, there is a set of nonzero vectors that produce zero output. These vectors can be added to any solution, and it’s still a solution. Therefore the solution is not unique. The space spanned by the set of vectors that produce zero is called the nullity of A. We would like to know a single solution and the characterize the nullity of A.
- When M = N and A is of full rank. We would like to find the unique solution x in a way that is efficient and accurate. In some cases the matrix A can be nearly singular. In such cases it can be impossible to compute the solution (even though it exists) because of numerical errors.
- When M > N , the system is over determined. In this case we would like to find the best compromise solution. Often, the best solution is defined in the sense of least squares. That is, minimize: (Ax − b)^2 (5) There are many ways to do this. One way is to convert the equations into another linear N × N problem that solves this least squares. This is
AT^ Ax = AT^ b (6)
These are called the normal equations of the initial problem. The solution is
x = (AT^ A)−^1 AT^ b (7)
The matrix (AT^ A)−^1 AT^ is called the psuedo inverse. Notice, sometimes (AT^ A)−^1 can be solved with a method for well-posed linear systems, but can be singular or nearly so. Therefore, the normal equations for overconstrained systems should be used with caution.
- In some cases we would like to solve
Ax = bi (8)
for many different bi’s
2 Mechanisms for Solving Linear Systems
For well-posed systems (i.e. square matrix of full rank) there are several mechanisms for solving. Note that explicitly computing the inverse A−^1 , is generally not recommended.
Gaussian Elimination: Somewhat robust. Can detect singularities. Not very efficient.
with corresponding wi’s that are zero gives Ax = 0. We know this because if multiply A by column of V , call it vi, we get
Avi = U W V T^ vi = U W (0,... , 1 ,... 0) = U (0,... , wi,... 0) = wiui, (12)
where is zero when wi = 0. Thus the null space is spanned by the vectors (columns of V T^ ) associated with zero wis is the null space. Any point in this space when multiplied by A returns 0. The space spanned by the vectors associate with the nonzero elements of W is called the range of A. The dimensionality of that space is the rank. The rank plus the nullity is equal to the size of A, which is N. Therefore, the nullity of A is the dimension of its null space.
Solving Singular Problems
If A is singular, one might want to single out a solution that is somehow better than the others. In this case want might want the smallest solution, i.e. the smallest length x^2. One way to do that is to use the diagonals, but replace 1/wj by zero where wj is zero. I.e. use zero whereever 1/wj blows up. The solution is then x = V [diag(1/wj )]
( U T^ b
) (13) This gives the shortest solution to the problem.
Proof
Consider a solution that is modified by a vector x′^ could it be shorter by some other solution? What happens when we add a vector x′^ that is in the null space.
|x + x′| =
∣∣ ∣V W −^1 U T^ b + x′
∣∣ ∣
=
∣∣ ∣W −^1 U T^ b + V T^ x′
∣∣ ∣
But, the first term has nonzero elements only in those places where the wj 6 = 0. The second term, because it’s in the null space, has nonzero elements only in those places where wj = 0. Therefore the two terms are orthogonal vectors, and their sum must be greater length than either part.
Solving Overconstrained Problems
We can solve over constrained problems and get the least squared solution. The solution strategy is the same
x = V [diag(1/wj )]
( U T^ b
) , (14)
but it is quaranteed to minimize the square of the residual
² = Ax − b. (15)
Proof
Consider the solution given above, and modify it by adding some arbitrary vector x′. Let b′^ = Ax′. Clearly b′^ is in the range of A.
|Ax − b + b′| =
∣∣ ∣
( U W V T^
) ( V W −^1 U T^ b
) − b + b′
∣∣ ∣
=
∣∣ ∣
( U W W −^1 U T^ − 1
) b + b′
∣∣ ∣
=
∣∣ ∣U
[( W W −^1 − 1
) U T^ b + U T^ b′
]∣∣ ∣
=
∣∣ ∣
( W W −^1 − 1
) U T^ b + U T^ b′
∣∣ ∣
However (W W −^1 − 1) is a diagonal matrix with nonzero element only where wj = 0. Because b′^ lies in the range of A, U T^ has nonzero elements only where wj 6 = 0. Therefore these terms are orthongal vectors, and the minimum of their sum is obtained when b′^ = 0.
3.2 Solving Homogeneous Problems
Consider a homogeneous problem of the form
Ax = 0. (16)
If it is overconstrained, we would like to solve the associated least squares problem, which minimizes min
[ ||²||^2
] where ² = Ax. (17) The solution of such systems is not unique—it is defined up to a scalar value. Therefore, we usually look for the solution to the constrained problem, i.e. ||x|| = 1. I.e. we look for the unit length solution that produces the smallest residual. The solution is given by SVD to be the column vector of V which corresponds to the singular value in W which has the smallest magnitude. How do we know this?
Proof
Consider the input s which is this vector, and assume it is position i. The output is
U W V T^ s = U W (0,... , 1 ,... 0) = U (0,... , wi,... 0) = wiui. (18)
Notice that ui is a column of U , and it has length 1. This is the shortest output possible from this matrix, because any other unit length input would include weighted sums of uis, with weights that must be larger than wi, because it is the smallest singular value.
