Download Statistical Hypothesis Testing: T-Test & Confidence Intervals for Mean Difference and more Slides Data Analysis & Statistical Methods in PDF only on Docsity!

Math 541: Statistical Theory II

Hypothesis Testing Based on Two Samples

Instructor: Songfeng Zheng

It is quite common to compare the properties of two distributions, for example, we would

like to see which distribution has a higher mean, or which distribution has a higher variance,

etc. This note describes how to conduct hypothesis testing regarding the mean and variance

when the two distributions under consideration are normal.

1 Comparing the Means of Two Normal Distributions

Let us consider a problem in which random samples are available from two normal distribu-

tions. The problem is to determine whether the means of the two distributions are equal.

Specifically, we assume that the random variables X 1

, · · · , X

m

form a random sample of size

m from a normal distribution for which both the mean μ 1

and the variance σ

2

1

are unknown;

and that the variables Y 1

, · · · , Y

n

form another independent random sample of size n from

another normal distribution for which both the mean μ 2

and variance σ

2

2

are unknown. In

this section, we will discuss several frequently seen cases.

1.1 The Case σ

2

1

= σ

2

2

= σ

2

We shall assume at this moment that the variance σ

2

1

= σ

2

2

= σ

2 is the same for both

distributions, even though the exact values are unknown.

Suppose it is desired to test the following hypotheses at a specified level of significance α

(0 < α < 1):

H

0

: μ 1

= μ 2

vs. H a

: μ 1

= μ 2

Intuitively, it makes sense to reject H 0

if

¯

X

m

−

¯

Y

n

is very different from zero, where

¯

X

m

and

¯

Y

n

are the means of the two samples, respectively. In spirit of the t test, we define

S

2

X

=

m ∑

i=

(X

i

−

¯

X

m

)

2

and S

2

Y

=

n ∑

j=

(Y

j

−

¯

Y

n

)

2

Then the test statistic we shall use is

T =

(m + n − 2)

1 / 2

(

¯

Xm −

¯

Yn)

(

1

m

+

1

n

) 1 / 2

(S

2

X

+ S

2

Y

)

1 / 2

Next, let us derive the distribution of T.

For each pair of values μ 1 and μ 2 , and for each σ

2

, the sample mean

¯

Xm has a normal

distribution with mean μ 1

and variance σ

2 /m, i.e.

¯

X

m

∼ N (μ 1

, σ

2 /m)

similarly

¯

Y

n

∼ N (μ 2

, σ

2

/n)

because both samples are from normal distribution. Furthermore,

¯

X

m

and

¯

Y

n

are indepen-

dent. It follows that the difference

¯

X

m

−

¯

Y

n

has a normal distribution with mean μ 1

− μ 2

and variance [(1/m) + (1/n)]σ

2 , i.e.,

¯

X

m

−

¯

Y

n

∼ N

(

μ 1

− μ 2

, [(1/m) + (1/n)]σ

2

)

.

Therefore, when the null hypothesis is true, i.e. μ 1

= μ 2

, the following random variable Z

will have a standard normal distribution:

Z =

¯

Xm −

¯

Yn

(

1

m

+

1

n

) 1 / 2

σ

∼ N (0, 1)

Also, for all values of μ 1

, μ 2

, and σ

2 , the random variable S

2

X

/σ

2 has a χ

2 distribution with

m − 1 degrees of freedom, and S

2

Y

/σ

2 has a χ

2 distribution with n − 1 degrees of freedom,

and the two random variables are independent. By the additive property of χ

2 distribution,

the following random variable W has a χ

2

distribution with m + n − 2 degrees of freedom:

W =

S

2

X

σ

2

+

S

2

Y

σ

2

=

S

2

X

+ S

2

Y

σ

2

∼ χ

2

m+n− 2

Furthermore, the four random variables

¯

X

m

,

¯

Y

n

, S

2

X

, and S

2

Y

are independent. This is

because: (i)

¯

X

m

and S

2

X

are functions of X 1

, · · · , X

m

; while

¯

Y

n

and S

2

Y

are functions

of Y 1

, · · · , Y

n

; and we know that X 1

, · · · , X

m

and Y 1

, · · · , Y

n

are independent. Therefore

{

¯

X

m

, S

2

X

} and {

¯

Y

n

, S

2

Y

} are independent. (ii) By the property of sample mean and sample

variance,

¯

Xm and S

2

X

are independent, and

¯

Yn and S

2

Y

are independent.

It follows that Z are W are independent. When the null hypothesis is true, i.e. μ 1

= μ 2

,

Z ∼ N (0, 1), and W ∼ χ

2

m+n− 2

. By the definition of t-distribution, we have the following

T =

(m + n − 2)

1 / 2

(

¯

X

m

−

¯

Y

n

)

(

1

m

+

1

n

) 1 / 2

(S

2

X

+ S

2

Y

)

1 / 2

=

¯ Xm−

¯ Yn

(

1

m

1

n

)

1 / 2

σ

[

(S

2

X

+S

2

Y

)/σ

2

m+n− 2

] 1 / 2

=

Z

[W/(m + n − 2)]

1 / 2

∼ t m+n− 2

Thus, to test the hypotheses

H

0

: μ 1

= μ 2

vs. H a

: μ 1

= μ 2

,

intuitively, if the null hypothesis was correct, the samples means

¯

X

m

and

¯

Y

n

should be very

close. Therefore, we can use T as the test statistic, and the distribution of the statistic T is

t m+n− 2

when H 0

is correct.

Now suppose we want to test the hypotheses at the significant level α, that is under the

null hypothesis μ 1

= μ 2

, the probability of rejecting H 0

is α. As we analyzed before, we

make our decision based on the difference of T from 0, that is if the value of |T | > c, we

will reject the null hypothesis, where c is a constant. Therefore the rejection probability is

P (|T | > c) = α where T ∼ t m+n− 2

. It is easy to calculate that the value of c is 100×(1−α/2)

percentile of tm+n− 2 distribution, tm+n− 2 (1 − α/2). Therefore, the final decision rule is:

if |T | > t m+n− 2

(1 − α/2), we would reject the null hypothesis at the significant level α;

otherwise, we fail to reject the null hypothesis.

Now let us consider the one-sided test

H

0

: μ 1

≤ μ 2

vs. H a

: μ 1

μ 2

.

In this case, we would reject the null hypothesis if the test statistic T is greater than a

predetermined positive number c. If the significant level is α, we can calculate the value

of c by P (T > c) = α where T ∼ t m+n− 2

. Then c = t m+n− 2

(1 − α). Therefore if the

alternative hypothesis is Ha : μ 1 > μ 2 , we can reject the null hypothesis if the test statistic

T > t m+n− 2

(1 − α).

The other case of one-sided hypothesis is

H

0

: μ 1

≥ μ 2

vs. H a

: μ 1

< μ 2

.

Similarly we would reject the null hypothesis if the test statistic T is less than a predetermined

negative number c. If the significant level is α, we can calculate the value of c by P (T <

c) = α where T ∼ t m+n− 2

. Then c = t m+n− 2

(α). Therefore if the alternative hypothesis is

H

a

: μ ≤ μ 0

, we can reject the null hypothesis if the test statistic T < t m+n− 2

(α).

1.2 The Case σ

2

2

= kσ

2

1

In the above, we discussed the case in which the two variances of the normal distributions are

the same although unknown. Now, let us generalize a little bit, suppose know that σ

2

2

= kσ

2

1

,

where k is a known constant. With these assumptions, we want to test the hypotheses

H 0 : μ 1 = μ 2 vs. Ha : μ 1 6 = μ 2.

For this problem, we still need a test statistic and the distribution of the statistic when the

null hypothesis is true. However, the statistic T is not appropriate here because T assumes

σ

2

1

= σ

2

2

. However, we can define another statistic similar to T as following:

T

′

=

(m + n − 2)

1 / 2

(

¯

Xm −

¯

Yn)

(

1

m

+

k

n

) 1 / 2

(S

2

X

+

S

2

Y

k

)

1 / 2

Let us find out the distribution of T

′ when the null hypothesis is true.

Similar as before,

¯

X

m

−

¯

Y

n

∼ N (μ 1

− μ 2

, σ

2

1

/m + σ

2

2

/n) = N (μ 1

− μ 2

, [(1/m) + (k/n)]σ

2

1

).

Therefore, when the null hypothesis is true, i.e. μ 1

= μ 2

, the following random variable Z

will have a standard normal distribution:

Z =

¯

X

m

−

¯

Y

n

(

1

m

+

k

n

) 1 / 2

σ 1

∼ N (0, 1)

Also, the random variable S

2

X

/σ

2

1

has a χ

2 distribution with m − 1 degrees of freedom, and

S

2

Y

/σ

2

2

has a χ

2

distribution with n − 1 degrees of freedom, and the two random variables

are independent. By the additive property of χ

2 distribution, the following random variable

W has a χ

2

distribution with m + n − 2 degrees of freedom:

W =

S

2

X

σ

2

1

+

S

2

Y

σ

2

2

=

S

2

X

+ S

2

Y

/k

σ

2

1

∼ χ

2

m+n− 2

Z and W are independent due to the same reason as before. By the definition of t-

distribution, we have

T

′

=

(m + n − 2)

1 / 2 (

¯

X

m

−

¯

Y

n

)

(

1

m

+

k

n

) 1 / 2

(S

2

X

+

S

2

Y

k

)

1 / 2

=

¯ Xm−

¯ Yn

(

1

m

k

n

)

1 / 2

σ 1

[

(S

2

X

+S

2

Y

/k)/σ

2

1

m+n− 2

] 1 / 2

=

Z

[W/(m + n − 2)]

1 / 2

∼ t m+n− 2

As we can see, if k = 1, the statistic T

′ will reduce to T. After we figure out the test statistic

T

′

and the null distribution, we can design a testing procedure as before.

1.3 Test H

0

: μ

1

− μ

2

= λ vs. H

a

: μ

1

− μ

2

= λ

In the above, we only checked the two means are equal or not. We can generalize the above

discussion a step further, i.e. to test

H 0 : μ 1 − μ 2 = λ vs. Ha : μ 1 − μ 2 6 = λ

where −∞ < λ < ∞ is a constant. As we can see, if λ = 0, this test reduces to the test we

discussed above.

To test these hypotheses, similar to the above discussion, when the variances of the two

normal distributions are equal, we can define the test statistic as

T =

(m + n − 2)

1 / 2

(

¯

Xm −

¯

Yn − λ)

(

1

m

+

1

n

) 1 / 2

(S

2

X

+ S

2

Y

)

1 / 2

.

Similarly, it can be shown that when the null hypothesis H 0

is true, T ∼ t m+n− 2

. We can

make our decision based on the value of the test statistic and the null distribution t m+n− 2

.

When the variances of the two normal distributions are not equal, but their relation is known

as σ

2

2

= kσ

2

1

, we can define the test statistic as

T

′

=

(m + n − 2)

1 / 2

(

¯

Xm −

¯

Yn − λ)

(

1

m

+

k

n

) 1 / 2

(S

2

X

+

S

2

Y

k

)

1 / 2

.

It can be shown that when the null hypothesis H 0 is true, T

′

∼ tm+n− 2. We can make our

decision based on the value of the test statistic and the null distribution t m+n− 2

.

Similarly, we can define the test procedure for the one-sided hypotheses.

1.4 Confidence Intervals for μ

1

− μ

2

As we studied before, the confidence intervals and the hypothesis testing process are equiv-

alent. Therefore, from the testing process, we can construct the 1 − α confidence intervals

for the difference of means of two normal distributions.

Firstly, under the case, σ

2

1

= σ

2

2

, the 1 − α confidence interval for μ 1

− μ 2

is

(

¯

X

m

−

¯

Y

n

) ± t

(

1 −

α

, m + n − 2

)

√

S

2

X

+ S

2

Y

m + n − 2

√

m

+

n

.

Indeed, in this case

h(X 1

, · · · , X

m

, Y

1

, · · · , Y

n

, μ 1

, μ 2

) =

(m + n − 2)

1 / 2

[(

¯

X

m

−

¯

Y

n

) − (μ 1

− μ 2

)]

(

1

m

+

1

n

) 1 / 2

(S

2

X

+ S

2

Y

)

1 / 2

is the pivot to construct the confidence interval.

Similarly, under the case, σ

2

2

= kσ

2

1

, the 1 − α confidence interval for μ 1

− μ 2

is

(

¯

X

m

−

¯

Y

n

) ± t

(

1 −

α

, m + n − 2

)

√

S

2

X

+ S

2

Y

/k

m + n − 2

√

m

+

k

n

.

and, in this case

h(X 1

, · · · , X

m

, Y

1

, · · · , Y

n

, μ 1

, μ 2

) =

(m + n − 2)

1 / 2 [(

¯

X

m

−

¯

Y

n

) − (μ 1

− μ 2

)]

(

1

m

+

k

n

) 1 / 2

(S

2

X

+ S

2

Y

/k)

1 / 2

is the pivot to construct the confidence interval.

2 Comparing the Variances of Two Normal Distribu-

tions

Assume that the random variables X 1

, · · · , X

m

form a random sample of size m from a

normal distribution for which both the mean μ 1

and the variance σ

2

1

are unknown; and that

the variables Y 1

, · · · , Y

n

form another independent random sample of size n from another

normal distribution for which both the mean μ 2 and variance σ

2

2

are unknown. In this

section, we will develop testing procedure to compare the two variances.

2.1 F Distribution

We first define and discuss the properties of a probability distribution family, called the

F distribution (Fisher-Snedecor distribution). This distribution arises in many important

problems of testing hypotheses in which two or more normal distributions are to be compared

on the basis of random samples from each of the distribution.

Consider two independent random variables Y and W , such that Y has a χ

2 distribution

with m degrees of freedom, and W has a χ

2 distribution with n degrees of freedom, where

m and n are given positive integers. Define a new random variable X as follows:

X =

Y /m

W/n

=

nY

mW

.

Then the distribution of X is called an F distribution with m and n degrees of freedom.

It can be proved that the if a random variable X has an F distribution with m and n degrees

of freedom, i.e. X ∼ F (m, n), then its p.d.f. is as follows:

f (x) =







Γ(

m+n

2

)m

m/ 2 n

n/ 2

Γ(

m

2

)Γ(

n

2

)

x

(m/2)− 1

(mx+n)

(m+n)/ 2

, x > 0

0 , x ≤ 0

Please note that when we speak of an F distribution with m and n degrees of freedom, the

order in which the numbers m and n are given is important. As we can see from the definition

or the p.d.f. of X, when m 6 = n, the F distribution with m and n degrees of freedom and

the F distribution with n and m degrees of freedom are two different distributions. In fact,

from its definition, it is straightforward to see that if X ∼ F (m, n), then its reciprocal 1/X

will have an F (n, m) distribution.

From the definitions of t distribution and F distribution, it is easy to see that if X ∼ t(n),

then X

2

∼ F (1, n). Indeed, since X ∼ t(n), then X could be written as X =

Z √

Y /n

, where

Z ∼ N (0, 1) and Y ∼ χ

2

n

. Then X

2

Z

2

Y /n

=

Z 2 / 1

Y /n

, where Z2 = Z

2 ∼ χ

2

1

, by the definition of

F random variable, X

2 ∼ F (1, n).

2.2 Comparing the Variances of Two Normal Distributions

Now let us go back to the problem of comparing the variances of the two normal distributions

proposed at the beginning of this section. Suppose that finally the following hypotheses are

to be tested at given level of significance α,

H

0

: σ

2

1

= σ

2

2

vs. H a

: σ

2

1

= σ

2

2

.

As in the previous section, we define

S

2

X

=

m ∑

i=

(X

i

−

¯

X

m

)

2

and S

2

Y

=

n ∑

j=

(Y

j

−

¯

Y

n

)

2

.

As proved before, S

2

X

/(m − 1) and S

2

Y

/(n − 1) are unbiased estimators of σ

2

1

and σ

2

2

, respec-

tively. Intuitively, it makes sense that if the null hypothesis was true, the ratio

V =

S

2

X

/(m − 1)

S

2

Y

/(n − 1)

should be close to 1. Therefore, we would reject the null hypothesis if the test statistic V is

too small or too big.

Under the null hypothesis, σ

2

1

= σ

2

2

,

V =

S

2

X

/(m − 1)

S

2

Y

/(n − 1)

=

[S

2

X

/σ

2

1

]/(m − 1)

[S

2

Y

/σ

2

2

]/(n − 1)

.

It is well-known that

S

2

X

σ

2

1

∼ χ

2

m− 1

and

S

2

Y

σ

2

2

∼ χ

2

n− 1

,

by the definition of F distribution, V ∼ F (m − 1 , n − 1).

By our intuitive decision rule, we would reject H 0

either V ≤ c 1

or V ≥ c 2

, where c 1

and c 2

are two constants such that P (V ≤ c 1

) + P (V ≥ c 2

) = α. The most convenient

choice of c 1 and c 2 is the one that makes P (V ≤ c 1 ) = P (V ≥ c 2 ) = α/2. That is,

choose c 1

= F (α/ 2 , m − 1 , n − 1), the α/2 percentile of F (m − 1 , n − 1), and choose c 2

=

F (1 − α/ 2 , m − 1 , n − 1), the 1 − α/2 percentile of F (m − 1 , n − 1).

This procedure can also be generalized to test the hypotheses

H

0

: σ

2

1

= rσ

2

2

vs. H a

: σ

2

1

= rσ

2

2

,

where r is a positive constant, and we notice that when r = 1 this will be reduced to the

previous case. Under this new null hypothesis, the value of V would be close to r, therefore,

V /r would be close to 1. Let us have a close look at the statistic V /r:

V

r

=

S

2

X

/(m − 1)

rS

2

Y

/(n − 1)

=

[

S

2

X

σ

2

1

]

/(m − 1)

r

[

S

2

Y

σ

2

1

]

/(n − 1)

=

[S

2

X

/σ

2

1

]/(m − 1)

[S

2

Y

/σ

2

2

]/(n − 1)

.

It follows that the statistic V /r ∼ F (m − 1 , n − 1) for the same reason as above.

With the test statistic V /r and the sampling distribution of the statistic under H 0 , we

would reject the null hypothesis at the significant level α if V /r ≤ c 1

or V /r ≥ c 2

, with

c 1

= F (α/ 2 , m − 1 , n − 1) and c 2

= F (1 − α/ 2 , m − 1 , n − 1).

Now, let us consider the one-sided hypotheses

H

0

: σ

2

1

≤ σ

2

2

vs. H a

: σ

2

1

σ

2

2

.

As before, we use the ratio

V =

S

2

X

/(m − 1)

S

2

Y

/(n − 1)

Since S

2

X

/(m − 1) and S

2

Y

/(n − 1) are unbiased estimators of σ

2

1

and σ

2

2

, respectively. The

data evidence prefers H a

if the ratio V is significantly big. Given a significant level α,

we would reject the null hypothesis H 0

if V > c where c is a constant, and we choose

c = F (1 − α, m − 1 , n − 1).

2.3 Confidence Intervals for the Ratio σ

2

1

/σ

2

2

As we studied before, the confidence intervals and the hypothesis testing process are equiv-

alent. Therefore, from the testing process, we can construct the 1 − α confidence intervals

for the ratio of variances of two normal distributions.

Let r = σ

2

1

/σ

2

2

, then equivalently, we are looking for a 1 − α confidence intervals for r. As

proved in last subsection, we know V /r ∼ F (m − 1 , n − 1), therefore we can design the pivot

as

h(X 1

, · · · , X

m

, Y

1

, · · · , Y

n

, r) =

V

r

=

S

2

X

/(m − 1)

rS

2

Y

/(n − 1)

∼ F (m − 1 , n − 1).

Choose c 1

= F (α/ 2 , m − 1 , n − 1) and c 2

= F (1 − α/ 2 , m − 1 , n − 1), then

P (c 1

≤ V /r ≤ c 2

) = 1 − α,

from which we can obtain the desired confidence interval as

V

F (1 − α/ 2 , m − 1 , n − 1)

≤

σ

2

1

σ

2

2

≤

V

F (α/ 2 , m − 1 , n − 1)

.

3 Exercises

Exercise 1. Assume that the random variables X 1

, · · · , X

m

form a random sample of size m

from a normal distribution for which the mean μ 1 is unknown and the variance σ

2

1

is known;

and that the variables Y 1

, · · · , Y

n

form another independent random sample of size n from

another normal distribution for which the mean μ 2

is unknown and variance σ

2

2

is known.

Please develop and testing procedure to test the following hypotheses at the significance level

α:

H 0 : μ 1 = μ 2 vs. Ha : μ 1 6 = μ 2.

Using the relation between hypothesis testing and confidence interval, identify the 1 − α

confidence interval for μ 1

− μ 2

.

Exercise 2. Let X 1 , · · · , Xm be a random sample from the exponential density with mean

θ 1

, and let Y 1

, · · · , Y

n

be a random sample from the exponential density with mean θ 2

. The

density function for exponential distribution is

f (x|θ) =

θ

e

−x/θ

; x > 0

a. Define an F -test procedure test the hypotheses

H 0 : θ 1 = θ 2 vs. Ha : θ 1 6 = θ 2.

[Hint: Find the connection between

∑ m

i=

X

i

and a χ

2 random variable. Do the same thing

to

∑ n

i=

Yi]

b. Generalize the above testing procedure to test the hypotheses

H 0 : θ 1 = rθ 2 vs. Ha : θ 1 6 = rθ 2 ,

where r is a positive number.

c. Find a 1 − α confidence interval for θ 1 /θ 2.