Download lab manual for communication and more Exams Communication in PDF only on Docsity!
DISCRETE-TIME RANDOM PROCESS (1)
DIGITAL SIGNALS: DETERMINISTIC AND RANDOM
Discrete-time (or digital) signal: Deterministic and random signals:
RANDOM VARIABLES
Single Random Variables
- Definition of random variables:
- Probability distribution function and probability density function:
- Mean (or mean value, or expected value):
- Variance:
- Moments:
Multiple Random Variables
- Joint distribution function and joint density function:
- Joint moments:
- Correlation and covariance of multiple random variables:
- Correlation coefficient:
- Relationship between random variables (Independence, Uncorrelatedness, and Orthogonality):
Prediction and Estimation
- Parameter estimation, bias and consistency: Sample mean and sample variance
- Linear Mean-Square Estimation: Orthogonality principle
Two Types of Random Variables: Uniform and Gaussian Random Variables
- Uniform random variable:
- Gaussian random variable:
Some useful MATLAB functions:
DISCRETE-TIME RANDOM PROCESS
DISCRETE-TIME (DIGITAL) SIGNALS: DETERMINISTIC AND RANDOM
Discrete-time (digital) signal: x ( n )= xa ( nTs ), (1) that is a function of an integer-valued variable n and may result from sampling a continuous-time (analog) signal xa ( t ) with an A/D (analog-to-digital) converter having a sampling interval Ts (or sampling rate f (^) s = 1 / Ts ) so that continuous time t becomes discrete-time nTs , i.e., t = nTs. For a digital signal x ( n ), therefore, time instant n means n Ts. The signals that we study here are always digital.
Deterministic and random signals:
Example 1. Measurements of signals using a digital oscilloscope Digital oscilloscopes are a commonly-used instrument for various signal measurements, and they are usually equipped with an averaging function that is used to reduce noise when measuring a noisy deterministic signal that is a desired deterministic signal interfered with unwanted noise.
(a) Measurements of an analog sinusoidal signal xa ( t )= sin( 2 π tT ) where T is the period.
When xa ( t )is recorded with the sampling interval Ts = T 20 , it has a digital form of x ( n )= sin( π n / 10 ). Repeating the measurement N times, we will get N sinusoidal waveforms xk ( n ) (where k = 1, 2, …, N ) that all are exactly the same, i.e., xk ( n )= sin( π n / 10 ) for all k , and the mean-value waveform (^1) ( ) sin( / 10 ) 1
x n n N
N k k =^ π =
is the same as all the N waveforms. At any time instant n , we can determine the
signal value x ( n ) in a definite way, e.g., for n =1 and n = 5 we have x ( n )=sin( π/ 10 ) and x ( n )=1,
respectively. These show that a deterministic signal x ( n ) is repeatable and predictable. (b) Measurements of thermal noise xa ( t ) , a random signal.
The thermal noise voltage generated in a resistor is a commonly-encountered random signal. When the noise signal xa ( t )is recorded by a digital oscilloscope, it has a digital form x ( n ). Repeating the record N times, the N recorded waveforms xk ( n ) (where k = 1, 2, …, N ) will be different and the mean-value waveform (^) =
N k k^
x n N (^) 1
(^1) ( ) approaches to zero when N →∞, which demonstrates that x ( n ) is not repeatable.
At time instant n the value of x ( n ) it may take one of the N possible waveforms xk ( n )for k = 1, 2, …, N. This shows that x ( n ) is not predictable. Thus, a random signal x ( n ) is unrepeatable and unpredictable.
The characteristics of deterministic and random signals can, in general, be summarized as follows:
- A deterministic signal can be described by a mathematical expression and reproduced exactly with repeated measurements; it is related to one signal.
- A random signal that is associated with a set of (digital) signals is not repeatable and predictable; thus it may only be described probabilistically or in terms of average behaves of the signal set.
RANDOM VARIABLES
Single Random Variables
Definition of random variables: A random variable x is defined when each elementary event
ω i in the sample space Ω , i.e., ω i ∈Ω, is mapped to a value of
x , and is assigned with a certain probability (Fig. 2). The sample space is defined by the set of all possible experimental outcomes (elementary events). An elementary event is the event consisting of a single element. A random variable x may be of discrete type or continuous type.
Example 2. Discrete random variables In the experiment of rolling a fair die, we have six elementary
events ω i ( i = 1, 2, … 6), and we assign to the six events ω i the
random variable x = x ( ω i )= i. Thus, the random variable x may
take one of the six numbers
x ( ω 1 )=1, x ( ω 2 )=2, … , x ( ω 6 )=
that are mapped from the six elementary events ω i. The random variable is of discrete type. The sample
space contains six elementary events, i.e., Ω ={ ω 1 , ω 2 , …, ω 6 }. The probability that each elementary event
may happen is 1/6, and is mathematically expressed as Pr{ x = i } = 1/6, where i =1, 2, …, 6 Example 3. Continuous random variables In the case of a uniform distribution over interval ( b , c ), a random variable x is of continuous type and
assumes a continuum of values in the interval ( b , c ). The sample space is Ω = { ω : b < ω< c }. There are an
infinite number of elementary events, each taking a value in the interval ( b , c ). The probability of each elementary event is
Pr{α ≤ x ≤α+∆α}≅∆ α( c − b ), where b < α < c and∆ α >0 is a small change, and ∆limα → 0 Pr{⋅}= 0.
Probability distribution function and probability density function: A random variable x is characterized statistically or described probabilistically by means of the
probability distribution function , Fx ( α), given by
Fx (α )= Pr{ x ≤ α }, (2)
which is a probability that x assumes a value that is less than α. A random variable can also be characterized statistically by the probability density function , f (^) x ( α),
given by f (^) x (α )= dFx (α) d α (3)
Fig. 2. Definition of a discrete random variable
We shall say that the statistics of a random variable x are known if we can determine the probability distribution function or the probability density function.
Example 4. Probability distribution function and probability density function (i) For the experiment of rolling a die in which the probability that the random variable assumes one of the six numbers k =1, 2, …, 6 is Pr{ x = k }= 1 / 6 , the distribution function and the density function of the discrete
random variable are, respectively,
α
α
α
α
Fx α , (^)
6 1
( ) ( )^1
k x (^) d Fx k
f d α δ α
(ii) For the uniform distribution, the distribution function and density function of the continuous random variable are, respectively,
c
b b c c b
b Fx
= =^ − < <
0, otherwise
( ) ( )^1 , b c f (^) x dd Fx c b
α^ α
Ensemble Averages of random variables
A random variable x can also be characterized by ensemble averages, e.g., mean and variance. Mean (or mean value, or expected value):
For a discrete random variable x that assumes a value of α k with probability Pr{ x = α k }, the mean
(expected value) is defined as = = = k
E { x } mx α k Pr{ x α k } (6)
For a continuous random variable x , this expectation may be written in terms of the probability density
function f x ( α)as
∞
E { x }= mx = −∞α fx (α) d α (7)
The expectation is a linear operation since E { ax + by } = aE { x } + bE { y } (8)
For a complex random variable, the mean-square value is E { zz *} = E {| z |^2 }. (9)
Variance:
For a discrete random variable x that assumes a value of α k with probability Pr{ x = α k }, the variance is
defined as = = { [^ − ]}^ =[ − ] = k
Var{ x } σ x^2 E x E { x }^2 α k E { x }^2 Pr{ x α k } (10a)
The n th moment of a random variable is defined by
∞ −∞
E { x n^ } = α nfx (α) d α (16)
which is the expected value of x n. The first order moment, obviously, is the mean value m (^) x. The mean-square value is E { x^2 }, the second moment, that is an important statistical average and often used
as a measure for the quality of an estimate.
The n th central moment of a random variable is defined by
∞ −∞
E {( x − mx ) n }= (α − mx ) nfx (α) d α (17)
which is the expected value of ( x − mx ) n. The variance σ x^2 is the second central moment.
The n th absolute moment and absolute central moment of a random variable are defined, respectively, as E {| x | n }and E {| x − mx | n } (18) Note that for a complex variable z , the second central moment E {( z − m z )^2 }is different form the second absolute central moment E {| z − mz |^2 }= E {( z − mz )( z − m z )*}, where *^ means the conjugate of a complex
value.
Multiple Random Variables
When a problem is involved with two or more random variables, it is necessary to study the statistical dependencies (or relations) that may exist between the random variables. The statistical dependencies can be joint distribution and density functions, and correlation and covariance.
Joint distribution function and joint density function: Given two random variables x ( 1 ), and x ( 2 ), the joint distribution function for the random variables is
F x ( 1 ), x ( 2 )(α 1 ,.α 2 )= Pr{ x ( 1 )≤α 1 , x ( 2 )≤ α 2 } (19)
and the joint density function is
( , ) ( 1 ),( 2 )( 1 , 2 ) 1 2
2
f x ( 1 ), x ( 2 )α 1 α 2 ∂α∂α Fx x α α
For a complex random variable z = x + jy the distribution function for z is given by the joint distribution
function
Fz ( α )=Pr{ x ≤ a , y ≤ b }, where α = a + jb (21)
Joint moments: Like in the case of a single random variable, the statistical dependencies can also be described by ensemble averages. The joint moments are just used for this purpose. The correlation and covariance of two random variables are the two joints most often used in this course. The joint moment of the random variables x and y is defined by E { xk^ y * l } (22) where y *is the conjugate of y.
The joint central moment of the random variables x and y is defined by E {( x − mx ) k ( y − my )* l } (23)
The orders of the moments are k+l.
Correlation and covariance of two random variables: Correlation : rxy = E { xy^ *}, (24)
that is a second order joint moment Covariance : c (^) xy = Cov( x , y )= E {[^ x − mx ][^ y − my ]*}^ = E { xy^ *}^ − mxm * y = rxy − mxm * y , (25)
that is the second order joint central moment. The correlation and covariance are used to statistically characterize the relationship between two random variables, and they play an important role in studying signal modeling, spectrum estimation, and Wiener filters.
Correlation coefficient: The correlation coefficient , a normalized covariance,
x y
xy x y x y xy cxy r m^ m
−^ *
An interesting property of the correlation coefficient is that
ρ xy ≤ 1 , or cxy ≤ σ x σ y (27)
Relationship between random variables (Independence, Uncorrelatedness, and Orthogonality): Two random variables x and y are said to be statistically independent if
f x , y (α ,β)= fx (α) fy ( β ) (28)
Two random variables x and y are said to be statistically uncorrelated if E { xy *}= E { x } E { y *} or rxy = mxm * y (29)
which is a weaker form of the independence. In this case, the covariance is zero, c (^) xy = rxy − mxm * y^ = 0. Thus, two random variables x and y will be s tatistically uncorrelated if their covariance is zero, c (^) xy = 0.
Note that statistically independent variables are always uncorrelated, but the converse is not true in general. A useful property of uncorrelated random variables is the following Var{ x + y }=Var{ x }+Var{ y } (30) since Var{ x + y }= E {[ x + y − mx − my ][ x + y − mx − my ]*^ } =Var{ x }+ cxy +( c (^) xy ) *+Var{ y }. Two random variables x and y are orthogonal if their correlation is zero, E { xy *}= 0 or rxy = 0.
Prediction and Estimation
Prediction and estimation are two general classes of problems encountered in statistical investigations. In the prediction case, the probabilistic model (e.g., a certain distribution, or density, function) of the problem is assumed to be known, and predictions are made concerning future observations. For example, in
Example 9. The bias and consistency of the sample mean (Example 3.2.3 on p. 74)
Let x be a random variable with a mean m x and variance σ 2 x. Given N uncorrelated observations of x that
are denoted by xn , the sample mean (^)
N x (^) n n x N m 1
ˆ (^1) has the expected value
{ } { } (^) x
N
n
x
N
n
E mx = N Exn = N m = m
= 1 = 1
which is an unbiased estimator. The variance of the sample mean is
{ } { } N x N N E x m N
x N m x x
N
n
n
N
n
n x
N
n
n
N
n
x n^2 1
2 1
2
1 1
Var ˆ Var Var =^1 Var =^ σ
= ^ −
= = = =
which goes to zero as N →∞. Therefore, the sample mean is an unbiased and consistent estimator.
Linear Mean-Square Estimation: The linear mean-square estimator y ˆ^ of a random variable y in terms of a random variable x is of the form, y ˆ^ = ax + b (37)
and the goal is to find the values for a and b that minimizes the mean-square error
ξ = E {( y − y ˆ)^2 } = E {( y − ax − b )^2 }. (38)
To minimize ξ , we differentiate ξ with respect to a and b and set zeros to the derivatives as follows,
( )^2 = { 2 ( − − )(− )} =− 2 [ { } + { 2 }+ ] = 0
a E a y ax b E y ax b x Exy aEx bmx
( )^2 = { 2 ( − − )(− 1 )} =− 2 [ + + ] = 0
∂ (^) y ax b E y ax b m am b b
E
b y x
Solving the two equations for a and b , we find
x
y
a xy σ
= ρ , and b = my − amx (41)
and further the optimum linear estimator for y is
x x^ y
y
y ˆ = xy σ ( x − m )+ m
Inserting y ˆ^ into ξ = E { ( y −ˆ y )^2 }, we find the minimum mean-square error of the estimator
ξ min = σ^2 y ( 1 − ρ^2 xy ) (43)
In one extreme case when x and y are uncorrelated, we have ρ xy = 0 so that a = 0 and b = E { y }. Thus, the
estimator for y is y ˆ^ = E { y } and ξ min = σ^2 y , which shows that the best estimator with the minimum mean-
square error ξmin = σ^2 y is the mean of y , and also shows that x is not used in the estimation of y , so that
knowing x does not affect the estimate of y. In another extreme case when | ρ xy |= 1 , ξ min = 0 , and it follows
that y = ax + b. The random variables x and y are linearly related to each other.
Two notes from this example: (i) The correlation coefficient provides a measure of the linear predictability between random variables. The
closer | ρ xy |is to 1, the more accurate the estimate y ˆ^ is to the random variable y.
(ii) The orthogonality principle : E { ( y − y ˆ) x } = E { ex } = 0 (where e = y − y ˆis called estimation error that is a random variable, and E { ex }is just the correlation between e and x , namely, E { ex } = rex ), which follows from ∂ξ ∂ a = E { 2 ( y − ax − b )(− x )} = − 2 E { ( y − y ˆ) x } = 0. This principle states that for the optimum linear predictor the estimation error will be orthogonal to the data x because of E { ex } = rex =0. It is fundamental in mean-square estimation problems.
Two Types of Random Variables: Uniform and Gaussian Random Variables
Uniform and Gaussian random variables are two types of important random variables in probability theory. Gaussian random variables are also called normal random variables. The random processes that are made up of a sequence of such random variables play an important role in statistical signal processing.
Uniform random variable: The statistical properties of a uniform random variable, i.e., its probability, probability distribution and probability density functions, its mean and variance, have been investigated in Examples 3-6, respectively.
Gaussian random variable: The density function of a Gaussian random variable x is of the form
= ^ − −
2
2 2 exp ( ) 2
( )^1
x
x x
x f^ m
We can, from the definitions, find that the mean E { x } equals mx and the variance Var ( x ) equals σ x^2. This
reveals that the density function of a Gaussian random variable is completely defined once the mean and the variance are given. The joint density function of Gaussian random variables x and y is given by
2 2 2
2 , 2
exp ( ) 2 ( )( ) 2 1
( , )^1
y
y x
x y xy x
x x y xy
xy f m m m^ m
Gaussian random variables have a number of important properties as follows: Property 1. If x and y are jointly Gaussian, then for any constants a and b the random variable z = ax + by is Gaussian with mean mz = amx + bm y , and variance
σ z^2^ = a^2 σ x^2 + b^2 σ y^2 + 2 ab σ x σ y ρ xy.
Property 2. If two jointly Gaussian random variables x and y are uncorrelated, ρ xy =0, then they are
statistically independent, f x , y (α ,β)= fx (α) fy ( β).
Property 3. If x and y are jointly Gaussian random variables then the optimum nonlinear mean-square estimator for y , y ˆ^ = g ( x ), that minimizes the mean-square error ξ= E { ( y − g ( x ))^2 } is a linear estimator y ˆ = ax + b
