Download Covariance Matrices in EECS 501: Gaussian PDF Applications & Properties and more Study notes Electrical and Electronics Engineering in PDF only on Docsity!
EECS 501 COVARIANCE MATRICES Fall 2001 DEF: A random vector is a vector of random variables ~x = [x 1... xN ]′. Note: Unless otherwise stated, a random vector is a column vector. DEF: The mean vector of random vector ~x is ~μ = E[~x] = [E[x 1 ]... E[xN ]]′.
DEF: The covariance matrix Kx = Λx of ~x is the N × N matrix whose (i, j)th^ element (Kx)ij = λxixj = E[xixj ] − E[xi]E[xj ]. Note: Kx = E[(~x − E[~x])(~x − E[~x])′] = E[~x~x′] − E[~x]E[~x]′^ (outer products). Also Outer product ~x~y′^ = [xiyj ] = N × N matrix having rank 1. Note: Inner product ~x′~y =
xiyi=scalar=Trace of outer product.
- Kx is a symmetric matrix: (Kx)ij = λxixj = λxj xi = (Kx)ji.
- Kx is a positive semidefinite matrix: For any vector ~a, the scalar ~a′Kx~a =
∑N
i=
∑N
j=1 ai(Kx)ij^ aj^ ≥^ 0.
- In particular, the diagonal elements of Kx have (Kx)ii = σ^2 xi ≥ 0. This is necessary but not sufficient for Kx to be positive semidefinite.
Thm: Let random vector ~y = A~x +~b for any constant matrix A and vector ~b. A need not be square. Then E[~y] = AE[~x] + ~b and Ky = AKxA′. Proof: Ky = E[(~y − E[~y])(~y − E[~y])′] = E[A(~x − E[~x])(A(~x − E[~x]))′] = E[A(~x − E[~x])(~x − E[~x])′A′] = AKxA′^ using (A~x)′^ = ~x′A′. #2: Define rv y = ~a′~x =
∑N
i=1 aixi. Then^ σ
2 y =^ ~a
′Kx~a ≥ 0.
DEF: Kx has N eigenvalues λi and associated eigenvectors vi which solve Kxvi = λivi, i = 1... N. Fact: Kx real & symmetric→ λi & vi real. Fact: Kx is positive semidefinite iff λi ≥ 0 , i = 1... N. Matlab: eig→ λi, vi.
Thm: Let V = [v 1 |v 2 |... |vN ] (matrix of eigenvectors) and ~y = V ′~x. Then: λyiyj = E[yiyj ] − E[yi]E[yj ] = λiδij = 0 if i 6 = j. Proof: Kxvi = λivi → KxV = V diag[λi] → Ky = V ′KxV = diag[λi] since v~i′^ v~j = 0 if i 6 = j (V is a unitary matrix: V ′V = V V ′^ = I). Note: This is called decorrelating or (pre)whitening the vector ~x. It is an essential part of communications and signal processing in noise.
DEF: Cross-correlation matrix Kxy = E[(~x − E[~x])(~y − E[~y])′]. Kxy = K yx′. Props: Kx+y = Kx + Ky + Kxy + Kyx = Kx + Ky + Kxy + K xy′ symmetric.
~z = A~x + ~b → Kzy = AKxy and Kyz = KyxA′. Compare to σ x^2 +y. DEF: ~x and ~y are uncorrelated if Kxy = [0] ↔ E[~x~y′] = E[~x]E[~y]′.
EECS 501 APPLICATIONS OF COVARIANCE MATRICES Fall 2001
- Let ~x =
x 1 x 2 x 3
(^). E[~x] =
(^). Kx =
Note: Kx symmetric (obvious), positive semidefinite (check: Matlab’s “eig”). Q: Kx has λ 3 = 0 and v 3 = [1, − 2 , 1]′. Significance of 0 eigenvalue? A: Let y = v 3 ′~x = 1x 1 − 2 x 2 + 1x 3. Then σ y^2 = v 3 ′Kxv 3 = v′ 3 λ 3 v 3 = 0. y = x 1 + x 3 − 2 x 2 = E[y] = v′ 3 E[~x] = 0 with probability 1. Not very random vector: x 1 = 2, x 2 = 3 → x 3 = 4 with probability 1!
- Suppose eigenvalues are 100,98,95,2,1,0.1. Significance of grouping? ~y = V ′~x → ~x = V ~y =
∑N
i=1 yivi^ where^ yi^ uncorrelated and^ σ
2 yi =^ λi. DEF: This is the finite-dimensional Karhunen-Loeve expansion of ~x. Idea: Since σ y^2 i ≈ 0 for i = 4, 5 , 6, approximate yi ≈ E[yi] for i = 4, 5 , 6. i.e.: Treat y 1 , y 2 , y 3 as uncorrelated rvs; y 4 , y 5 , y 6 as known constants. Point: Have compressed data [x 1... x 6 ]′^ to [y 1 , y 2 , y 3 ]′; reduced dimension.
EECS 501 MULTIDIMENSIONAL GAUSSIAN PDF Fall 2001
DEF: {x 1... xN } are jointly Gaussian rvs (JGRV) if their joint pdf is f~x( X~) = (^) (2π)N/ 2 √^1 |det K x|^
exp[− 12 ( X~ − ~μ)′K x− 1 ( X~ − μ~)]. ~x ∼ N (~μ, Kx).
f~x( X~)dX 1... dXN = 1: See p.250. V =matrix of eigenvectors.
- ~y = V ′~x → f~y (Y~ ) = (^) |det V^1 | f~x( X~ = V Y~ ) so integrates to 1. |det V | = 1. = (^) (2π)N/ 2 √|^1 det V ′K xV^ |^
exp[− 12 (Y~ − E[~y])′diag[ (^) λ^1 i ](Y~ − E[~y])]
=
∏N
i= √^1 2 πλi^ exp[−^
1 2 (Yi^ −^ E[yi])
(^2) /λi] = ∏N i=1 fyi (Yi)
Point: For {x 1... xN } JGRV, uncorrelated↔independent. Unusual! JGRV {x 1... xn} have diagonal Kx → {x 1... xN } independent rvs.
- Any linear combination of JGRV is JGRV. {xi} Gaussian 6 = {xi} JGRV.
- f~x,~y ( X, ~ Y~ ) above form→ ~x and ~y each JGRV; f~x|~y ( X~|Y~ ) above form.
- 2-D: fx,y (X, Y ) = (^2) πσ^1 xσy
1 −ρ^2 exp
[
− (^) 2(1−^1 ρ (^2) ) ( X
2 σ^2 x^ +^
Y 2 σ y^2 −^
2 ρXY σxσy )
]
where K[x,y]′ =
[
σ x^2 λxy λxy σ y^2
]
[
σx 0 0 σy
] [
1 ρxy ρxy 1
] [
σx 0 0 σy
]
