Download Linear Algebra: Vector Spaces, Linear Dependence, Subspaces, and Transformations - Prof. E and more Study notes Aerospace Engineering in PDF only on Docsity!
- Background Linear Algebra -
Eugene M. Cliff
August 24, 1999
1 Introduction
These notes are intended to supplement the treatment in the Gill-Murray & Wright text [1]. While the discussion in the text is mostly adequate for our needs, it does present some unfortunate limitations in its outlook. We shall present some material to broaden the approach. Our presentation is heavily influenced by Luenberger [3]. There are a number of excellent texts for most of this material (e.g. [2]). Numerical aspects of linear algebra are in the recent book by Trefethen and Bau [4].
2 Vector Spaces
A vector space is a set of objects (the vectors), the real (or complex) field of scalars and two operations connecting them. The operations are: vector addition and multiplication by scalars. Associated with any two vectors is a (unique) third vector - their sum. Also, given a vector and a scalar there is a unique vector - the product. The operations must enjoy certain properties (vector addition is commutative and associative). These are generally stated in the form of certain axioms that the vector operations must satisfy ( see [3], pp 11-12). Among these axioms is the requirement that the set contain a unique vector 0 , the additive identity element (meaning that v + 0 = v, for all elements). The usual ‘vectors’ in Rn^ are the most common example. Addition means add component-wise. Somewhat more abstractly, we also mention the space of continuous functions on the interval [0, 1]. Vector-addition and scalar multiplication are defined in the obvious way. To be rigorous one should
prove the ‘closure’ property, e.g. that the sum of two continuous functions is a continuous function.
2.1 Linear Combinations
Given a finite set of vectors, and a corresponding set of scalars we can form a ‘new’ vector by the operation
vnew = α 1 v 1 + α 2 v 2 + · · · + αpvp
In this case we say that vnew is a linear combination of the underlying vectors.
2.2 Linear Dependence
Strictly said, linear dependence is best thought of as a property of certain finite-sets. That is, given a finite set of vectors, say S ≡ {v 1 , v 2 ,... , vp} we say that S is a linearly dependent set iff there is a set of scalars (at least one such set), not all zero, such that the corresponding linear com- bination is the zero vector (i.e.
∑p i=1 αivi^ =^0 ).^ A set of vectors that is not linearly dependent is said to be linearly independent. Note that if S is a linearly independent set and if
∑p i=1 αıvı^ =^0 , then necessarily all of the scalars (αı) must be zero. With this definition, any set that includes the zero-vector is linearly de- pendent. For another example of a linearly dependent set, consider the vector space R^2 and the set of vectors given by
S ≡ {
In this case the scalars (α 1 = 1, α 2 = 1, α 3 = −1) demonstrate the depen- dence.
2.3 Subspaces
It happens that certain subsets of vectors have the ‘closure’ property, meaning that addition or scalar multiplication of anything in this set produces another element of the set. Such sets are called subspaces - they are vector spaces in their own right. It’s clear that any subspace must contain the zero vector. We agree that the set consisting of only the zero vector is a subspace and
members until we get down to a linearly independent set. If we throw out any more vectors the span of the thus-reduced set will not be the same as the span of the original. A spanning set that is linearly independent is said to be a basis for the subspace. It turns out that the number of elements in a basis is a property of the subspace - this is the dimension of the subspace. Again, it requires proof to show that if we start with two sets with the same span, this process of throwing out results in linearly independent sets with the same number of elements. Our description characterizes a basis as a minimal spanning set. If we throw out any more elements then the span of the resulting set is decreased. It’s sometimes useful to think of starting with a small set and adding vectors
- to produce the desired spanning set. In this view if we add a vector to a basis and keep the same spanning set, it’s clear that the new set is not linearly independent. Hence one can also think of a basis as the largest linearly independent set (with a given span). Any basis has a naturally associated subspace; namely the space spanned by the basis. While the basis is not unique (a given subspace has many bases) the number of elements in any such basis is always the same. As noted above, this number is the dimension of the subspace. When adding subspaces we have dim[(U + V)] ≤ dim[U] + dim[V], with equality iff the sum is direct.
2.7 A Vector and Its Representation
Once we have a basis then any vector can be uniquely described by providing the scalar components in its representation. That is, if
vsample = α 1 v 1 + α 2 v 2 + · · · + αn, vn
then the n-tuple of scalars (α 1 , α 2 ,... , αn) represent the vector vsample in terms of the given basis. Perhaps the canonical example of a n-dimensional space is Rn, wherein the vectors are these n-tuples of scalars. In other examples the basis is fixed so that we can blur this distinction and think of the collection of scalars as the vector. In most cases this causes no harm, but we should be aware of it. To make this more concrete, consider the space of continous functions on the unit interval, and the subspace spanned by the set of vectors S = {cos(·), sin(·)}.
Since the two vectors are linearly independent, the set S is a basis for this two-dimensional subspace. Now however, there should be a clear distinction between the pair (α, β) and the function f (t) = α cos(t)+β sin(t). In fact, we derive alot of benefit from the fact that one can blur the distinction between f (·) and the pair (α, β).
3 Transformations and Functionals
One of the things we do with spaces (or their subspaces) is to define certain related maps or functions. Suppose X and Y are two vector spaces, then a rule that associates a unique element of Y to every element of X is a transformation or a map. In some settings this idea must be generalized so that the rule is defined for only some proper subset D ⊂ X. We write this as T : D ⊂ X 7 → Y
A special case of particular interest occurs when the image space Y is the scalar field; such a transformation is called a functional.
3.1 Linear Transformations
Linearity of maps is a natural idea in the vector space setting; it means that applying the transformation to a linear combination of vectors, produces the same result as applying the transformation to each component vector and then forming the linear combination. In symbols, we have
T (
∑^ p
i=
αiui) =
∑^ p
i=
αiT (ui).
In the case when Y is the (real or complex) field we speak of a linear func- tional.
3.2 Representation of a Linear Transformation
Suppose we are in the situation T : X 7 → Y, that dim[X] = n, dim[Y] = m and that we have bases, say U, and V, for X and Y, respectively. Then for any x ∈ X, we have x =
αu, so that
T (x) = T (
αu) =
αT (u).
- ‖u + v‖ ≤ ‖u‖ + ‖v‖
- ‖αv‖ = |α| ‖v‖
A vector space and an associated norm are called a normed space. A commonly cited non-Euclidean example is C[0, 1] - the space of real-valued continuous functions on the interval [0, 1]. The vector space structure has been noted above and the norm in this case is given by ‖v‖ ≡ max 0 ≤t≤ 1 |v(t)|. The notation C[0, 1] means the vector space with this norm. Note that we
can define a different norm on the same vector space (e.g. ‖v‖ ≡
0 |v(t)|^ dt. The normed space with this integral norm is different from C[0, 1].
4.1 Inner Products
It turns out that there is a way to generalize the Euclidean case that will produce a special class of norms and normed spaces. An inner-product is a complex (possibly real) valued function that assigns to any pair of vectors a scalar value - their inner (or dot) product. We shall write this as: < u, v >. Again, we require certain axioms to end up with a useful concept
- < u, v >= < v, u > (overbar means complex conjugate)
- < (u 1 + u 2 ), v > = < u 1 , v > + < u 2 , v >
- < αu, v > = α < u, v >
- < v, v > ≥ 0 and < v, v > = 0 iff v = 0.
With these properties it turns out that ‖v‖ ≡
< v, v > is a norm. The vec- tor space X, together with the inner-product define an inner-product space (sometimes called a pre-Hilbert space). Here again, the most common exam- ple is the Euclidean case:
< (α 1 , α 2 ,... , αn), (β 1 , β 2 ,... , βn) > =
i=1,n
αi βi = αT^ β.
Note we have mixed in some matrix notation here. A common non-Euclidean example is the space of continuous functions along with the inner-product < u, v >≡
0 u(t)^ v(t)^ dt.
4.2 Orthogonality
Just as in the Euclidean case we say that two vectors are orthogonal if the inner-product is zero. In this case one can easily show the Pythagorean Theorem holds: ‖(u + v)‖^2 = ‖u‖^2 + ‖u‖^2. The idea of orthogonality of two vectors can be extended by saying that a vector is orthogonal to a set S iff it is orthogonal to every element of the set. For given S the collection of all vectors orthogonal to the given set is called the orthogonal complement of S ( we say S-perp and write S⊥). In fact, one can prove that S⊥^ is a subspace (even if S isn’t). If we start with a set that is a subspace S ⊂ X, then we have a neat decomposition:
S
S⊥^ = X.
This means that any vector in X can be uniquely decomposed into a part that’s in S and a second part in its orthogonal complement. In the plane we picture a line through the origin (S) and a second line at right-angles to it (S⊥). In three-dimensions we might have a plane for (S) and a perpendicular line for the orthogonal complement. Finally, if S is a subspace then S⊥⊥
(meaning [S⊥] ⊥ ) gets us back to S.
4.3 Adjoints and Transposes
Suppose we have two inner-product spaces (X and Y) and a linear map between them T : X 7 → Y. For a given x ∈ X and y ∈ Y we can compute the real-number < T x, y >Y, the inner-product in the Y-space. Now we get a little wierd. For the given T and fixed y suppose we want to evaluate the inner-product for a variety of vectors x ∈ X. We complain that this is alot of work: first map the new x to Y by computing T (x), and then compute the Y-space inner-product. For this fixed T and y is there an element in X that will work with the X-space inner-product? We are asking for an element x∗^ ∈ X so that
< x, x∗^ >X = < T x, y >Y ∀x ∈ X.
In fact, this requirement defines a new linear operator T ∗^ : Y 7 → X. T ∗ called the adjoint of the operator T. In the real-Euclidean case, where T is represented by a matrix M the calculation looks like:
< T x, y >Y = < x, x∗^ >X (M x)T^ y = xT^ (MT^ y).
References
[1] Practical Optimization, Gill, P.E., Murray, W. and Wright, M.H., Aca- demic Press, 1982
[2] Finite-dimensional vector spaces, Halmos, Paul R., Van Nostrand, 1958, QA261 H33 1958
[3] Optimization by vector space methods, Luenberger, David G., Wiley, 1969, QA402.5 L
[4] Numerical Linear Algebra, Trefethen, L.N., and Bau, D., SIAM, Philadel- phia, QA184 T74, 1997
