Chapter 4 Jointly distributed Random variables, Slides of Linear Algebra

Download Chapter 4 Jointly distributed Random variables and more Slides Linear Algebra in PDF only on Docsity!

Chapter 4

Jointly distributed Random variables

Continuous Multivariate distributions

Continuous Random Variables

Definition: Joint Probability density function

Two random variable are said to have joint probability density function f ( x, y ) if

1. 0  f  x y , 

2. f  x , y  d xd y 1

 

   

3. P  (^)  X Y , (^)   A    f (^)  x y , (^)  dxdy A

Joint Probability Density Function (pdf)

Definition: Marginal Density

Let X and Y denote two RVs with joint pdf f ( x,y ), then the marginal density of X is

and the marginal density of Y is

Definition: Conditional Density

Let X and Y denote two RVs with joint pdf f ( x,y ) and marginal densities f (^) X ( x ), f (^) Y ( y ), then the conditional density of Y given X = x and the conditional density of X given Y = y are given by

f X  x  f  x , y  d y



 

 (^) 

f Y  y  f  x , y  d x



 

Y X X

f x y f y x f x

X Y Y

f x y f x y f y

Marginal and Condition Density

The bivariate Normal distribution

Sir Francis Galton (1822 –1911, England)

Let the joint distribution be given by:

2 2 1 1 1 1 2 2 2 2 1 1 2 2 (^1 2 )

2

, 1

x x x x

Q x x

        



 (^)               (^)   ^         ^ ^ ^  ^ ^ ^    

 1 2  1 , 2 (^1 2 ) 1 2

, e

Q x x

f x x



where

This distribution is called the bivariate Normal distribution.

The parameters are  1 ,  2 ,  1 ,  2 and 

The properties of this distribution were studied by Francis Galton and discovered its relation to the regression, term Galton coined.

The bivariate normal distribution

Surface Plots of the bivariate Normal

distribution

The bivariate normal distribution

Note: We can have a more compact joint using linear algebra:

(1) Determine the inverse and determinant of Σ (the covariance matrix)

The bivariate normal distribution

 

  

     





 





 

  

  

  

  

 

 



( )' ^ ( ) 2

1 exp 2 (| |)

1

( ) 2 ( )( ) ( ) 2 ( 1 )

1 exp 2 ( 1 )

1 ( , )

1 1 / 2

2 2 2

2 2 2

2 2 1

(^211) 1

1 1 2 2 1 2

1 2

x μ x μ 



 

 

  



  

x x x x f x x

 2 12 1

21

2 1 2

2 2 2

2 2 1 2

2 1

2 (^212) 2

2 1

2 12

2 2

2 2 1 12 2

21

2 1

 

 

  

     

 

Let MGF of a bivariate normal is given by:

Note: When ρXY = 0 –i.e., X and Y are independent. The MGF is:

The bivariate normal distribution: MGF

( 2 )]

( , ) exp[ 12

2 2 2

2 2

mXY t 1 t 2  t 1  X  t 2  Y  t 1  X  t  Y   XYtt  X  Y

( )]

( , ) exp[ 2 2 2

2 2

mXY t 1 t 2  t 1  X  t 2  Y  t 1  X  t  Y

f 1 (^)  x 1 (^)  f (^)  x 1 (^) , x 2 (^)  dx 2







Marginal distributions for the Bivariate Normal

Recall the definition of marginal distributions for continuous RV:

and

In the case of the bivariate normal distribution the marginal

distribution of xi is Normal with mean  i and standard deviation  i.

f 2 (^)  x 2 (^)  f (^)  x 1 (^) , x 2 (^)  dx 1







Proof:

The marginal distributions of x 2 is

f (^) 2  x 2 (^)  f (^)  x 1 (^) , x 2 (^)  dx 1





 (^)   

 1 2  1 , 2 2 1 1 2

e

Q x x

dx

   

 





where

 

2 2 1 1 1 1 2 2 2 2 1 1 2 2 1 2 2

2

, 1

x x x x

Q x x

        



 (^)     (^)    (^)    (^)     (^)   ^         ^ ^  ^ ^ ^    

Now:

2 2 1 1 1 1 2 2 2 2

1 1 2 2 (^1 2 )

2

, 1

x x x x

Q x x

 (^)                 (^)   ^         ^ ^ ^  ^ ^ ^    

(^2 2 ) 1 1

x a x a a

c x c

b b b b

     

2 1 1 2 2 2 2 2 1 2 1

x x x

   

  ^  

 

 

 

 

 

2 2 1 2 2 2 2 2 2 2 1 2 2 1 2 1 2

^ x^^  x          

Marginal distributions for the Bivariate Normal

Hence (^)   2 2 2 b  1  or b  1 1 



Also    

1 2 2 (^2 2 )

a x

b

       

 

 

 

1 2 1 2 2

x

        

and

 

1 1 2 2

a x

    

Marginal distributions for the Bivariate Normal

Summarizing

2 2 1 1 1 1 2 2 2 2

1 1 2 2 (^1 2 )

x x x x

Q x x

   ^      

 ^ ^ ^ ^ ^  

2

x 1 a

c

b

where 2 b   1 1 

 

1 1 2 2

a x

    

2 2 2

2

x

c





and

Marginal distributions for the Bivariate Normal

Thus f 2  x 2  f  x 1 , x 2  dx 1





 (^) 

 1 2  (^1) , 2 2 1 1 2

1 e 2 1

Q x x dx    

 (^) 



 



2 (^11) 2 2 1 1 2

1 e 2 1

x a c b dx    

 (^)      (^)   (^)      



 



2 2 1 1 2 2 1 1 2

2 1 e 2 1 2

c^ x^ a be (^) b dx b

   ^    



 



2 2 2 2

1 2

2

1

2

x e

   

    (^)    ^ 

Marginal distributions for the Bivariate Normal

• Thus the marginal distribution of x 2 is Normal with mean  2 and

standard deviation  2.

• Similarly, the marginal distribution of x 1 is Normal with mean  1 and

standard deviation  1.

Note: This derivation is much easier using MGFs.

Use the MGF of a bivariate normal. To get the MGF of the marginal of

X, set t 2 =0.

( )] ( )

( , 0 ) exp[

( 2 )]

( , ) exp[

1

2 2 1 1 1

12

2 2 2

2 2 1 2 1 2 1

m t t t m t

m t t t t t t tt

XY X X X

XY X Y X Y XY X Y

 

      

Marginal distributions for the Bivariate Normal

Bivariate Normal Distribution with marginal distributions

Marginal distributions: Bivariate Normal

 

2 (^11) 2 1 2^1

x a b

f x x e

 b

    (^)    

Hence 

Then, the conditional distribution of x 2 given x 1 is Normal with mean and standard deviation:

  and

1 1 2^1 2 2

a x

     

2 b   (^) 1 2   1 1 

Conditional distributions: Bivariate Normal

• Bivariate Normal Distribution with conditional distribution

Conditional distributions: Bivariate Normal

21

1 1 | 2 11 12 22

2 2

1 1 | 2 1 12 22 ( )

    

    

   x 

• Using matrix notation, the conditional moments are given by:

x 2

x 1

Regression to the mean (μ 1 = μ 2 = μ) :

Major axis of ellipses

( 1 1 ) 1

2 2 | 1 2  

    x

Note: ( ) ( ) ( 1 1 ) 2 1

12 1 1 2 1

2

1 2

12 2 | 1 2  

   x x

Conditional distributions: Bivariate Normal

 (^21) | ( x 1 ); 0  1

28

• The Kalman filter (KF) uses the observed data to learn about the unobservable state variables, which describe the state of the model.
• KF models dynamically what we measure, zt , and the state, yt. In the simple, linear model we have:

yt = A yt-1 + wt ( state or transition equation )

zt = H yt + v (^) t ( measurement equation )

wt , v (^) t : error terms, with zero mean and variance Q and R , respectively.

• Based on time t-1 information, the KF generates predictions for yt : yt|t-1 = A yt-1|t-1 + B ut P t|t-1 = A P t-1 A T^ + Q (conditional variance of yt )
• It also generates an update, once the information t is known: yt|t = yt|t-1 + P t|t-1 H T^ ( F t|t-1)-1^ et|t- P t|t = P t|t-1 – P t|t-1 H T^ ( F t|t-1)-1^ H P t|t-

Conditional distributions: KF Application