Download Probability Theory - Lecture Notes | STAT 541 and more Study notes Biostatistics in PDF only on Docsity!
3. Probability Theory
3.1 Basic Ideas, Definitions, and Properties
POPULATION = Unlimited supply of five types of fruit, in equal proportions. O 1 = Macintosh apple O 2 = Golden Delicious apple
… …
… …
… …
O 3 = Granny Smith apple
O 4 = Cavendish banana O 5 = Plantain banana
Experiment 1: Randomly select one fruit from this population, and record its type.
Sample Space: The set S of all possible elementary outcomes of an experiment.
S = { O 1 , O 2 , O 3 , O 4 , O 5 } #( S ) = 5
Event: Any subset of a sample space S. (“Elementary outcomes” = simple events .)
A = “Select an apple.” = { O 1 , O 2 , O 3 } #( A ) = 3 B = “Select a banana.” = { O 4 , O 5 } #( B ) = 2
Event P (Event)
A 3 5 = 0.
B
2 5 = 0. 5 5 = 1.
P ( A ) = 0.6 “The probability of randomly selecting an apple is 0.6.” As # trials → ∞ P ( B ) = 0.4 “The probability of randomly selecting a banana is 0.4.”
trials of
experiment
#(Event)
trials
0
1
A
B 1/
1/
2/ 2/
... ...
e.g., (^) A B A A (^) B A...
1/
2/3 3/ 3/
4/
1/
(^1 2 3 4 )...
General formulation may be facilitated with the use of a Venn diagram:
Sample Space: S = { O 1, O 2 , …, Ok } #( S ) = k
O 1
O 2
O 4
... Om Ok
O 3
A O
m +1 Om +
Om +
Experiment ⇒
Event A = { O 1 , O 2 , …, Om } ⊆ S #( A ) = m ≤ k
Definition: The probability of event A , denoted P ( A ), is the long-run relative frequency with which A is expected to occur, as the experiment is repeated indefinitely.
Fundamental Properties of Probability
For any event A = { O 1 , O 2 , …, Om } in a sample space S ,
- 0 ≤ P ( A ) ≤ 1
2. P ( A ) = 1 2
1
( (^) i ) ( ) ( ) ( ) ( (^) m )
m
i
P O P O P O P O P O
=
∑ =^ +^ +^ +^ …+
Special Cases:
- P (∅) = 0
- P ( S ) = = 1 “certainty” 1
( (^) i )
k
i
P O
=
∑
- If all the elementary outcomes of S are equally likely , i.e.,
P ( O 1 ) =^ P ( O 2 ) = … =^ P ( Ok ) =
k , then… #( ) ( ) #( )
A m P A k
S
.
Example: P ( A ) =
5 = 0.6,^ P ( B ) =
(2) A ∩ B = “ A and B ” = {All outcomes in S that A and B share in common.}
intersection
= {All outcomes that result when events A and B occur simultaneously .}
Example: A ∩ B = “Select a 2 and a ♣” = {2♣} ⇒ P ( A ∩ B ) =
Definition: Two events A and B are said to be disjoint , or mutually exclusive , if they cannot occur simultaneously, i.e., A ∩ B = ∅, hence P ( A ∩ B ) = 0.
S
A B
Example: A = “Select a 2” and C = “Select a 3” are disjoint events.
Exercise: Are A = {2 , 3 , 4 , 5 ,...}^4 4 4 4 and B ={2 , 3 , 4 , 5 ,...}^6 6 6 6 disjoint? If not, find A ∩ B.
(3) A ∪ B = “ A or B ” = {All outcomes in S that are either in A or B , inclusive.}
union
P ( A ∪ B ) = P ( A ) + P ( B ) − P ( A ∩ B )
= 0, if A and B are disjoint.
Example: A ∪ B = “Select either a 2 or a ♣” has probability
P ( A ∪ B ) =
52 −^
Example: A ∪ C = “Select either a 2 or a 3” has probability
P ( A ∪ C ) =
52 −^ 0 =^
Note: Formula (3) extends to n ≥ 3 disjoint events in a straightforward manner:
(4) P ( A 1 ∪ A 2 ∪ … ∪ An ) = P ( A 1 ) + P ( A 2 ) + … + P ( An ).
Question: How is this formula modified if the n events are not necessarily disjoint?
Example: Take n = 3 events…
S^ Then^ P ( A^ ∪^ B^ ∪^ C ) =
A
C
B
P ( A ) + P ( B ) + P ( C )
− P ( A ∩ B ) − P ( A ∩ C ) − P ( B ∩ C )
+ P ( A ∩ B ∩ C ).
Exercise: For S = {January,…, December}, verify this formula for the three events A = “Has 31 days,” B = “Name ends in r,” and C = “Name begins with a vowel.”
Exercise: A single tooth is to be randomly selected for a certain dental procedure. Draw a Venn diagram to illustrate the relationships between the three following events: A = “upper jaw,” B = “left side,” and C = “molar,” and indicate all corresponding probabilities. Calculate the probability that all of these three events, A and B and C , occur. Calculate the probability that none of these three events occur. Calculate the probability that exactly one of these three events occurs. Calculate the probability that exactly two of these three events occur. (Think carefully.) Assume equal likelihood in all cases.
incisors
The three “set operations” – union, intersection, and complement – can be unified via...
Exercise: Using a Venn diagram, convince yourself that these statements are true in general. Then verify them for a specific example, e.g., A = “Pick a picture card” and B = “Pick a black card.”
incisors
premolars
premolars
canine canine
canine canine
molars
( A ∪ B ) C^ = AC^ ∩ BC
( A ∩ B ) C^ = AC^ ∪ BC
DeMorgan’s Laws
3 5 7 9 11
1 36
6 36
1 36
2 36
2 36
3 36
3 36
4 36
4 36
5 36
5 36
Again, the probability distribution for X can be organized in a probability table , and displayed via a probability histogram , both of which enable calculations to be done easily:
x f ( x ) = P ( X = x )
Total Area = 1
2 3 4 5 6 7 8 9 10 11 12
P ( X = 7 or X = 11) Note that “ X = 7” and “ X = 11” are disjoint! = P ( X = 7) + P ( X = 11) via Formula (3) above = 6/36 + 2/36 = 8/
P (5 ≤ X ≤ 8)
= P ( X = 5 or X = 6 or X = 7 or X = 8) = P ( X = 5) + P ( X = 6) + P ( X = 7) + P ( X = 8) = 4/36 + 5/36 + 6/36 + 5/ = 20/
P ( X < 10) = 1 − P ( X ≥ 10) via Formula (1) above
= 1 − [ P ( X = 10) + P ( X = 11) + P ( X = 12)]
= 1 − [ 3/36 + 2/36 + 1/36 ] = 1 − 6/36 = 30/
Exercise: How could event E = “Roll doubles” be characterized in terms of a random variable? ( Hint : Let Y = “Difference between the two dice.”)
The previous example motivates the important topic of...
Discrete Probability Distributions
In general, suppose that all of the distinct population values of a discrete random variable X are sorted in increasing order: x
Total Area = 1
1 <^ x 2 <^ x 3 < …,^ with corresponding probabilities of occurrence f ( x (^) 1 ), f ( x 2 ), f ( x 3 ), … Formally then, we have the following.
Definition: f ( x ) is a probability distribution function for the discrete random variable X if, for all x , f ( x ) ≥ 0 AND all
x
∑^ f^ x = 1.
In this case, f ( x ) = P ( X = x ) , the probability that the value x occurs in the population.
The cumulative distribution function (cdf) is defined as, for all x ,
F ( x ) = P ( X ≤ x ) = all
i
i x x
f x ≤
∑ =^ f ( x^1 ) +^ f ( x^2 ) + … +^ f ( x ).
Therefore, F is piecewise constant , increasing from 0 to 1.
Furthermore, for any two population values a < b , it follows that
P ( a ≤ X ≤ b ) = (^) ∑ f ( ) x
b
a
= F ( b ) – F ( a −)
where a −^ is the value just preceding a in the sorted population.
Exercise: Sketch the cdf F ( x ) for Experiments 3a and 3b above.
| | | | X
x 1 x 2 x 3 … x …
F ( x 1 )
F ( x 2 )
F ( x 3 )
f ( x 1 )
f ( x 2 ) f ( x 3 )
f ( x ) … | | | | X
x 1 x 2 x 3 … x …
Experiment 4: At a party, guests randomly select one pastry from two trays
(continually refilled), where the distribution of calories X and X
30
30
60
30
90
60
120
90
150
90
120
150
Probability Histogram
X2 = Calories
20 40 60 80 100
30 60 90
Probability Histogram
X1 = Calories
80 100 120 140 160
90 150
1 2 are indicated below. Tray 1
Uniform Distribution
Probability Table
x f 1 ( x ) 90 1/ 120 1/ ¾ Mean( X 1 ) = μ 1 = ( 90 )(1/3) + ( 120 )(1/3) + ( 150 )(1/3) = 120 cals 150 1/ equally likely outcomes^ ¾^ Var( X^1 ) =^ σ^12 = (–30)^2 (1/3) + (0)^2 (1/3) + (30)^2 (1/3) = 600 cals^2
Tray 2
Probability Table
x f 2 ( x ) 30 3/ 60 2/6^ ¾^ Mean( X^2 ) =^ μ^2 = (^30 )(3/6) + (^60 )(2/6) + (^90 )(1/6) = 50 cals 90 1/6^ ¾^ Var( X 2 ) =^ σ 22 = (–20)^2 (3/6) + (10)^2 (2/6) + (40)^2 (1/6) = 500 cals^2
Summary (Also refer back to 2.4 - Summary)
POPULATION
Discrete random variable X
Probability Table → Probability Histogram
x f ( x ) = P ( X = x ) x 1 f ( x 1 ) x 2 f ( x 2 ) . . .
μ = E [ X ] = Σ x f ( x )
E [( X − μ) 2 ] = Σ ( x − μ) 2 f ( x )
σ 2 = or
E [ X^2 ] − μ^2 = Σ x^2 f ( x ) − μ^2
X
Parameters
SAMPLE, size n
Relative Frequency Table → Density Histogram
x (^) f ( x ) = freq ( x ) n x 1 f ( x 1 ) x 2 f ( x 2 ) . . .
xk f ( xk )
x =^ Σ^ x^ f ( x )
n
n − 1 Σ^ ( x^ −^ x^ )^
(^2) f ( x )
s^2 = or n
n − 1 [Σ^ x
2 f ( x ) − x 2 ]
X
Statistics
X and S^2 can be shown to be unbiased estimators of μ and σ 2 , respectively. That is, E ⎡⎣^ X ⎤⎦^ = μ, and E ⎡^ S^2 ⎤ = σ^2 ⎣ ⎦. (In fact, they are MVUE .)
Related (but not identical) to this is the idea that, of all linear combinations c x 1 1 (^) + c x 2 2 (^) + … + c xn n of the data { x 1 (^) , x 2 , … , xn }(such as X , with ) which are also unbiased , the one that minimizes MSE is called BLUE (B
c 1 (^) = c 2 (^) = … = cn =1/ n est Linear Unbiased Estimator). It can be shown that, in addition to being MVUE (as stated above), X is also BLUE. To summarize,
MVUE gives: Min Variance among all unbiased estimators
≤ Min Variance among linear unbiased estimators
= Min MSE among linear unbiased estimators (since MSE = Var + Bias^2 ),
given by BLUE (by def).
The Venn diagram below depicts these various relationships.
Unbiased (^) Linear
Minimum MSE
Minimum Variance
BLUE
MVUE
X
S^2 Minimum variance among linear unbiased estimators
Minimum variance among all unbiased estimators
Comment: If MSE → 0 as n → ∞ , then θˆis said to have mean square convergence
to θ. This in turn implies “ convergence in probability ” (via “Markov's Inequality,”
also used in proving Chebyshev’s Inequality), i.e., θ^ ˆ is a consistent estimator of θ.
