Covariance and Correlation: Proof and Identities, Study Guides, Projects, Research of Probability and Statistics

Download Covariance and Correlation: Proof and Identities and more Study Guides, Projects, Research Probability and Statistics in PDF only on Docsity! Covariance and Correlation Math 217 Probability and Statistics Prof. D. Joyce, Fall 2014 Covariance. Let X and Y be joint random vari- ables. Their covariance Cov(X, Y ) is defined by Cov(X, Y ) = E((X − µX)(Y − µY )). Notice that the variance of X is just the covariance of X with itself Var(X) = E((X − µX)2) = Cov(X,X) Analogous to the identity for variance Var(X) = E(X2)− µ2 X there is an identity for covariance Cov(X) = E(XY )− µXµY Here’s the proof: Cov(X, Y ) = E((X − µX)(Y − µY )) = E(XY − µXY −XµY + µXµY ) = E(XY )− µXE(Y )− E(X)µY + µXµY = E(XY )− µXµY Covariance can be positive, zero, or negative. Positive indicates that there’s an overall tendency that when one variable increases, so doe the other, while negative indicates an overall tendency that when one increases the other decreases. If X and Y are independent variables, then their covariance is 0: Cov(X, Y ) = E(XY )− µXµY = E(X)E(Y )− µXµY = 0 The converse, however, is not always true. Cov(X, Y ) can be 0 for variables that are not inde- pendent. For an example where the covariance is 0 but X and Y aren’t independent, let there be three outcomes, (−1, 1), (0,−2), and (1, 1), all with the same probability 1 3 . They’re clearly not indepen- dent since the value of X determines the value of Y . Note that µX = 0 and µY = 0, so Cov(X, Y ) = E((X − µX)(Y − µY )) = E(XY ) = 1 3 (−1) + 1 3 0 + 1 3 1 = 0 We’ve already seen that when X and Y are in- dependent, the variance of their sum is the sum of their variances. There’s a general formula to deal with their sum when they aren’t independent. A covariance term appears in that formula. Var(X + Y ) = Var(X) + Var(Y ) + 2 Cov(X, Y ) Here’s the proof Var(X + Y ) = E((X + Y )2)− E(X + Y )2 = E(X2 + 2XY + Y 2)− (µX + µY )2 = E(X2) + 2E(XY ) + E(Y 2) − µ2 X − 2µXµY − µ2 Y = E(X2)− µ2 X + 2(E(XY )− µXµY ) + E(Y 2)− µ2 Y = Var(X) + 2 Cov(X, Y ) + Var(Y ) Bilinearity of covariance. Covariance is linear in each coordinate. That means two things. First, you can pass constants through either coordinate: Cov(aX, Y ) = aCov(X, Y ) = Cov(X, aY ). Second, it preserves sums in each coordinate: Cov(X1 +X2, Y ) = Cov(X1, Y ) + Cov(X2, Y ) and Cov(X, Y1 + Y2) = Cov(X, Y1) + Cov(X, Y2). 1