Download Least Squares Approximation and Inner Products and more Study notes Calculus in PDF only on Docsity!
Inner Products
Adrian Down
March 07, 2006
1 Introduction
1.1 Motivation
Suppose we have some data points (xj , yj ), j ∈ { 1 ,... , n }, where n > 2. In general, these points cannot be interpolated by a single line of the form y = αx + β. Instead of finding an exact linear interpolation, we would like to find the “best” possible linear approximation. We define “best” to be the linear approximation that minimizes the sum of the error of the interpolation at each point,
s =
∑^ n
j=
(yj − αxj − β)^2
To find the desired linear approximation, our task is to determine the coef- ficients α and β.
1.2 Matrix notation
Recall the general form of a matrix equation,
Ax = b
x and b are vectors in Rn. A is an n × n matrix. A is said to be singular if the column space of A is not equal to Rn, i.e. the columns of A do not form a basis of Rn. If A is singular, Ax = b is not solvable for all b. With this notation, the least squares solution is the vector x such that |Ax − b|^2 is minimized, where
|x|^2 = x^21 +... + x^2 n
1.3 Continuous approximation
It is also possible to form a continuous “least squares” approximation to a function. Suppose, for example, that we would like to find a polynomial p(x) of degree n − 1 which interpolates a function f (x) on the interval [− 1 , 1]. In the continuos limit, the sum of the squares of the errors at each point becomes an integral,
s =
− 1
f (x) − p(x)
dx
Note. The discrete and continuous least squares approximation are connected by the notion of orthogonal projection.
2 Inner products
2.1 Properties
Rather than providing a static definition of the inner product, we consider the properties desired of an inner product, denoted by ·, on an abstract vector space. The requirements that we place on the inner product are as follows.
Binary operation · takes two vectors as inputs and produces a single number. Given any x and y in a vector space V , x · y is a number that satisfies certain properties.
Symmetry x · y = y · x
Bilinearity · is linear in each argument. Given scalars a and b,
(ax + by) · ζ = a(x · ζ) + b(y · ζ)
By symmetry, · is linear in the second argument as well.
Positive definiteness
x · x ≥ 0 ∀x and x · x = 0 ⇔ x = 0
Note. Equality occurs only when
0 = f (tm) = (tma − b) · (tma − b)
For an inner product of a vector with itself to equal 0, the vector must be equal to 0 by positivity. Hence, equality occurs only when,
b = tma
2.2.2 Triangle inequality
Definition (Length). We define the length of a vector x in a vector space V to be the square root of the square of the vector,
|x| =
x · x
Theorem.
|x + y| ≤ |x| + |y|
Proof. Consider the inner product of (x + y) with itself. Using symmetry and bilinearity,
(x + y) · (x + y) = x · x + 2x · y + y · y
Using the Cauchy-Schwartz inequality,
(x · y)^2 ≤ (x · x)(y · y)
Using this result,
x · y =
x · x
y · y (1) ⇒ (x + y)^2 ≤ x · x + 2
x · x
y · y + y · y (2)
x · x +
y · y
Taking the square root of 3,
|x + y| ≤ |x| + |y|
2.3 Polarization identity
This identity relates the general inner product of two vectors to their lengths, Theorem.
x · y =
|x + y|^2 − |x − y|^2
Proof. Using symmetry and bilinearity to expand the inner products in the identity,
|x + y|^2 − |x − y|^2 = (x + y) · (x + y) − (x − y) · (x − y)
x · x + 2x · y + (^) y · y − x · x + 2x · y − (^) y · y
= 4x · y
2.4 Vector space of functions
2.4.1 Definition
We would like to define an continuous inner product that holds on the vector space V of continuous functions on the interval [a, b]. The inner product takes the form
|f |^2 =
(^) b
a
w(x)f 2 (x)dx
where w(x) is a weighting function. With this definition, the functional inner product satisfies the properties described above. For example, using the polarization identity,
f · g =
(^) b
a
w(x)f (x)g(x)dx =
(^) b
a
w(x)
(f + g)^2 − (f − g)^2
dx
2.4.2 Examples
We could make a definition
|f |^2 = waf 2 (a) + wbf 2 (b) +
(^) b
a
w(x)f 2 (x)dx
From this definition, we could derive a new expression for f · g. We can also consider the effect of non-trivial weight functions.
