Covariance
When discussing a single RV, we used the notion of variance to cap-
ture how much that RV could differ from its expected value. With
two RVs we have a similar notion called the covariance, defined as
cov(X, Y ) = E[(X−E[X])(Y−E[Y])] = E[X Y ]−E[X] E[Y].
Note that cov(X, X) = var(X). It is straightforward to show that if
two RVs are independent, then cov(X, Y ) = 0.
The covariance gives a way to quantify how two variables move to-
gether when they are not independent. When cov(X, X)>0, then
Xand Ytend to have the same sign (i.e., one being positive indicates
that the other is likely positive). In contrast, when cov(X, X)<0,
then Xand Ytend to have the opposite sign (i.e., one being positive
indicates that the other is likely negative).
To show that the covariance actually quantifies something important,
remember our rule that var(X+Y) = var(X)+var(Y)only when X
and Yare independent? What about when the RVs are dependent?
In general, we can say that:
var(X+Y) = var(X) + var(Y) + 2cov(X, X ).
Correlation coefficient
The covariance discussed above seems like a very helpful way to quan-
tify the dependence between RVs. One glaring problem is that it’s
not normalized. While the covariance can be bounded using the in-
dividual variances, any bound derived in this way would depends on
the distribution of the variables (or even the scaling of the variables).
So, it’s hard to compare from one situation to the next.
60
ECE 3077 Notes by M. Davenport, J. Romberg and C. Rozell. Last updated 21:24, June 25, 2014
