Sufficient Statistics and Factorization Theorem in Mathematical Statistics - Prof. Jun Sha, Study notes of Mathematical Statistics

Download Sufficient Statistics and Factorization Theorem in Mathematical Statistics - Prof. Jun Sha and more Study notes Mathematical Statistics in PDF only on Docsity!

logo

Stat 709: Mathematical Statistics

Lecture 19

Jun Shao

Department of Statistics University of Wisconsin Madison, WI 53706, USA

logo

Lecture 19: Sufficient statistics and factorization

theorem

Data reduction without loss of information

A statistic T (X ) provides a reduction of the σ -field σ (X ) Does such a reduction results in any loss of information concerning the unknown population? If a statistic T (X ) is fully as informative as the original sample X , then statistical analyses can be done using T (X ) that is simpler than X. The next concept describes what we mean by fully informative.

Definition 2.4 (Sufficiency)

Let X be a sample from an unknown population P ∈ P, where P is a family of populations. A statistic T (X ) is said to be sufficient for P ∈ P (or for θ ∈ Θ when P = {Pθ : θ ∈ Θ} is a parametric family) iff the conditional distribution of X given T is known (does not depend on P or θ ).

logo

Remarks

Once we observe X and compute a sufficient statistic T (X ), the original data X do not contain any further information concerning the unknown population P (since its conditional distribution is unrelated to P) and can be discarded. A sufficient statistic T (X ) contains all information about P contained in X and provides a reduction of the data if T is not one-to-one. The concept of sufficiency depends on the given family P. If T is sufficient for P ∈ P, then T is also sufficient for P ∈ P 0 ⊂ P but not necessarily sufficient for P ∈ P 1 ⊃ P.

Example 2.

Suppose that X = (X 1 , ..., Xn) and X 1 , ..., Xn are i.i.d. from the binomial distribution with the p.d.f. (w.r.t. the counting measure)

fθ (z) = θ z^ ( 1 − θ )^1 −z^ I{ 0 , 1 }(z), z ∈ R, θ ∈ ( 0 , 1 ).

Consider the statistic T (X ) = (^) ∑ni= 1 Xi , which is the number of ones in X.

logo

Remarks

Once we observe X and compute a sufficient statistic T (X ), the original data X do not contain any further information concerning the unknown population P (since its conditional distribution is unrelated to P) and can be discarded. A sufficient statistic T (X ) contains all information about P contained in X and provides a reduction of the data if T is not one-to-one. The concept of sufficiency depends on the given family P. If T is sufficient for P ∈ P, then T is also sufficient for P ∈ P 0 ⊂ P but not necessarily sufficient for P ∈ P 1 ⊃ P.

Example 2.

Suppose that X = (X 1 , ..., Xn) and X 1 , ..., Xn are i.i.d. from the binomial distribution with the p.d.f. (w.r.t. the counting measure)

fθ (z) = θ z^ ( 1 − θ )^1 −z^ I{ 0 , 1 }(z), z ∈ R, θ ∈ ( 0 , 1 ).

Consider the statistic T (X ) = (^) ∑ni= 1 Xi , which is the number of ones in X.

logo

How to find a sufficient statistic?

Finding a sufficient statistic by means of the definition is not convenient It involves guessing a statistic T that might be sufficient and computing the conditional distribution of X given T = t. For families of populations having p.d.f.’s, a simple way of finding sufficient statistics is to use the factorization theorem.

Theorem 2.2 (The factorization theorem)

Suppose that X is a sample from P ∈ P and P is a family of probability measures on (Rn, Bn) dominated by a σ -finite measure ν. Then T (X ) is sufficient for P ∈ P iff there are nonnegative Borel functions h (which does not depend on P) on (Rn, Bn) and gP (which depends on P) on the range of T such that

dP d ν

(x) = gP

T (x)

h(x).

To prove Theorem 2.2, we need the following lemma whose proof can be found in the textbook.

logo

How to find a sufficient statistic?

Finding a sufficient statistic by means of the definition is not convenient It involves guessing a statistic T that might be sufficient and computing the conditional distribution of X given T = t. For families of populations having p.d.f.’s, a simple way of finding sufficient statistics is to use the factorization theorem.

Theorem 2.2 (The factorization theorem)

Suppose that X is a sample from P ∈ P and P is a family of probability measures on (Rn, Bn) dominated by a σ -finite measure ν. Then T (X ) is sufficient for P ∈ P iff there are nonnegative Borel functions h (which does not depend on P) on (Rn, Bn) and gP (which depends on P) on the range of T such that

dP d ν

(x) = gP

T (x)

h(x).

To prove Theorem 2.2, we need the following lemma whose proof can be found in the textbook.

logo

Lemma 2.

If a family P is dominated by a σ -finite measure, then P is dominated by a probability measure Q = (^) ∑∞ i= 1 ci Pi , where ci ’s are nonnegative constants with ∑∞ i= 1 ci = 1 and Pi ∈ P.

Proof of Theorem 2.

(i) Suppose that T is sufficient for P ∈ P. For any A ∈ Bn, P(A|T ) does not depend on P. Let Q be the probability measure in Lemma 2.1. By Fubini’s theorem and the result in Exercise 35 of §1.6,

Q(A ∩ B) =

∞

j= 1

cj Pj (A ∩ B) =

∞

j= 1

cj

∫

B

P(A|T )dPj

∫

B

∞

j= 1

cj P(A|T )dPj =

∫

B

P(A|T )dQ

for any B ∈ σ (T ). Hence, P(A|T ) = EQ (IA|T ) a.s. Q, where EQ (IA|T ) denotes the conditional expectation of IA given T w.r.t. Q.

logo

Lemma 2.

If a family P is dominated by a σ -finite measure, then P is dominated by a probability measure Q = (^) ∑∞ i= 1 ci Pi , where ci ’s are nonnegative constants with ∑∞ i= 1 ci = 1 and Pi ∈ P.

Proof of Theorem 2.

(i) Suppose that T is sufficient for P ∈ P. For any A ∈ Bn, P(A|T ) does not depend on P. Let Q be the probability measure in Lemma 2.1. By Fubini’s theorem and the result in Exercise 35 of §1.6,

Q(A ∩ B) =

∞

j= 1

cj Pj (A ∩ B) =

∞

j= 1

cj

∫

B

P(A|T )dPj

∫

B

∞

j= 1

cj P(A|T )dPj =

∫

B

P(A|T )dQ

for any B ∈ σ (T ). Hence, P(A|T ) = EQ (IA|T ) a.s. Q, where EQ (IA|T ) denotes the conditional expectation of IA given T w.r.t. Q.

logo

Proof of Theorem 2.2 (continued)

Let gP (T ) be the Radon-Nikodym derivative dP/dQ on the space (Rn, σ (T ), Q). Then

P(A) =

∫ P(A|T )dP =

∫ EQ (IA|T )dP =

∫ EQ (IA|T )gP (T )dQ

∫ EQ [IAgP (T )|T ]dQ =

∫ IAgP (T )dQ =

∫

A

gP (T ) dQ d ν

d ν

for any A ∈ Bn. Hence, dP d ν

(x) = gP

T (x)

h(x) (1)

holds with h = dQ/d ν.

(ii) Suppose that (1) holds. Then dP dQ

dP d ν

i= 1

ci

dPi d ν = gP (T )

i= 1

gPi (T ) a.s. Q, (2)

where the second equality follows from Exercise 35 in §1.6.

logo

Proof of Theorem 2.2 (continued)

Let gP (T ) be the Radon-Nikodym derivative dP/dQ on the space (Rn, σ (T ), Q). Then

P(A) =

∫ P(A|T )dP =

∫ EQ (IA|T )dP =

∫ EQ (IA|T )gP (T )dQ

∫ EQ [IAgP (T )|T ]dQ =

∫ IAgP (T )dQ =

∫

A

gP (T ) dQ d ν

d ν

for any A ∈ Bn. Hence, dP d ν

(x) = gP

T (x)

h(x) (1)

holds with h = dQ/d ν.

(ii) Suppose that (1) holds. Then dP dQ

dP d ν

i= 1

ci

dPi d ν = gP (T )

i= 1

gPi (T ) a.s. Q, (2)

where the second equality follows from Exercise 35 in §1.6.

logo

Proof of Theorem 2.2 (continued)

Let A ∈ σ (X ) and P ∈ P. The sufficiency of T follows from

P(A|T ) = EQ (IA|T ) a.s. P, (3)

where EQ (IA|T ) is given in part (i) of the proof. This is because EQ (IA|T ) does not vary with P ∈ P, and result (3) and Theorem 1.7 imply that the conditional distribution of X given T is determined by EQ (IA|T ), A ∈ σ (X ).

By (2), dP/dQ is a Borel function of T. For any B ∈ σ (T ), ∫

B

EQ (IA|T )dP =

∫

B

EQ (IA|T )

dP dQ

dQ

∫

B

EQ

IA

dP dQ

∣T

dQ =

∫

B

IA

dP dQ

dQ =

∫

B

IAdP.

This proves (3) and completes the proof.

logo

Exponential famlilies

If P is an exponential family, then Theorem 2.2 can be applied with

g θ (t) = exp{[ η( θ )]^ τ^ t − ξ ( θ )},

i.e., T is a sufficient statistic for θ ∈ Θ. In Example 2.10 the joint distribution of X is in an exponential family with T (X ) = (^) ∑ni= 1 Xi. Hence, we can conclude that T is sufficient for θ ∈ ( 0 , 1 ) without computing the conditional distribution of X given T.

Example 2.11 (Truncation families)

Let∫ φ (x) be a positive Borel function on (R, B) such that b a φ^ (x)dx^ <^ ∞^ for any^ a^ and^ b,^ −∞^ <^ a^ <^ b^ <^ ∞. Let θ = (a, b), Θ = {(a, b) ∈ R^2 : a < b}, and

fθ (x) = c( θ ) φ (x)I(a,b)(x), c( θ ) =

[∫ (^) b

a

φ (x)dx

]− 1

logo

Example 2.11 (continued)

Then {fθ : θ ∈ Θ}, called a truncation family, is a parametric family dominated by the Lebesgue measure on R. Let X 1 , ..., Xn be i.i.d. random variables having the p.d.f. f θ. Then the joint p.d.f. of X = (X 1 , ..., Xn) is

n

i= 1

f θ (xi ) = [c( θ )]nI(a,∞)(x( 1 ))I(−∞,b)(x(n))

n

i= 1

φ (xi ), (4)

where x(i) is the ith ordered value of x 1 , ..., xn. Let T (X ) = (X( 1 ), X(n)), g θ (t 1 , t 2 ) = [c( θ )]nI(a,∞)(t 1 )I(−∞,b)(t 2 ), and h(x) = (^) ∏ni= 1 φ (xi ). By (4) and Theorem 2.2, T (X ) is sufficient for θ ∈ Θ.

logo

Example 2.12 (Order statistics)

Let X = (X 1 , ..., Xn) and X 1 , ..., Xn be i.i.d. random variables having a distribution P ∈ P, where P is the family of distributions on R having Lebesgue p.d.f.’s. Let X( 1 ), ..., X(n) be the order statistics given in Example 2.9. Note that the joint p.d.f. of X is

f (x 1 ) · · ·f (xn) = f (x( 1 )) · · · f (x(n)).

Hence, T (X ) = (X( 1 ), ..., X(n)) is sufficient for P ∈ P.

The order statistics can be shown to be sufficient even when P is not dominated by any σ -finite measure, but Theorem 2.2 is not applicable (see Exercise 31 in §2.6).