Null Hypothesis Significance Testing: Understanding Type I and Type II Errors and Power, Slides of Statistics

Download Null Hypothesis Significance Testing: Understanding Type I and Type II Errors and Power and more Slides Statistics in PDF only on Docsity!

Null Hypothesis Significance

Testing

Null Hypothesis Significance Testing

• Decision to reject or fail to reject H (^) o
  • P value
  • Probability of obtaining the observed results if Ho is

true

• By convention, use the significance level of p <.
• Conclude that it is highly unlikely that we would

obtain these results by chance, so we reject Ho

• Caveat! The fact that there is a significance level does

not mean that there is a simple ‘yes’ or ‘no’ answer to

your research question

One and Two-tailed Tests

• One-tailed / Directional Test

– Run this when you have a prediction about the

direction of the results

• Two-tailed / Non-Directional Test

– Run this when you don’t have a prediction about

the direction of the results

Recall previous example…

• Research Qu

• Do anxiety levels of students differ from anxiety

levels of young people in general?

• Prediction

• Due to the pressure of exams and essays, students

are more stressed than young people in general

• Method

• You know the mean score for the normal young

population on the anxiety measure = 50

• You predict that your sample will have mean > 50
• Run a one-tailed one-sample t test at p < .05 level

Dilemma

• But! What if your prediction is wrong?

• Perhaps students are less stressed than the general

young population

• Their own bosses, summers off, no mortgages
• With previous one-tailed test, you could only reject Ho

if you got an extremely high sample mean

• What if you get an extremely low sample mean?

• Run a two-tailed test

• Hedge your bets
• Reject Ho if you obtain scores at either extreme of the

distribution, very high or very low sample mean

Two-tailed Test

• You will reject H (^) o when a score appears in the highest 2.5% of the distribution or the lowest 2.5%
• Note that it’s not the highest 5% and the lowest 5% as then you’d be operating at p =. level, rejecting Ho for 10% of the distribution
• So, we gain ability to reject Ho for extreme values at either end but values must be more extreme

Errors in NHST

• Remember we are dealing with probabilities

• We make our decision on the basis of the

likelihood of obtaining the results if Ho is true

• There is always the chance that we are making an

error

• Two kinds of Error

• We reject H (^) owhen it is true (Type I error)
  • We say there’s a significant difference when there’s not
• We accept H (^) o when it is false (Type II error)
  • We say there is no significant difference when there is

Type I Error

• Our anxiety example
• Predict that students will have greater anxiety score than young people in general
• Test H (^) o that students’ anxiety levels do not differ from young people
• One-tailed one sample t-test at p <.
• Compare sample mean with sampling distribution of mean for the population (H (^) o )

Type I Error

• For example, if p = .04, this means that there is a very small chance that your sample mean came from that population, - But this is still a chance, you could be rejecting Ho when it is in fact true
• Researchers are willing to accept this small risk (5%) of making a Type I error, of rejecting Ho when it is in fact true
• Probability of making Type I error = alpha α = the significance level that you chose - .05,.

Type II Error

• So why not set a very low significance level to

minimise your risk of making a Type I error?

– Set p < .01 rather than p <.

• As you decrease the probability of making a

Type I error you increase the probability of

making a Type II error

• Type II Error

– Fail to reject Ho when it is false

– Fail to detect a significant relationship in your data

when a true relationship exists Docsity.com

Four Outcomes of Decision Making

True State of Nature

Decision Ho is True H (^) o is False

Accept H (^) o Correct Decision Type II Error

Reject H (^) o Type I Error Correct Decision

Power

• You should minimise both Type I and Type II

errors

– In reality, people are often very careful about Type

I (i.e. strict about α) but ignore Type II altogether

• If you ignore Type II error, your experiment

could be doomed before it begins

– even if a true effect exists (i.e. H 1 is correct), if β is

high, the results may not show a statistically

significant effect

• How do you reduce the probability of a Type II

How do we increase the power of our

experiment?

• Factors affecting power
  • The significance level (α)
  • One-tailed v two-tailed test
  • The true difference between Ho and H 1 (μo - μ 1 )
  • Sample Size (n)

The Influence of α on Power

• Reduce the significance level (α)…

• Reduce the probability of making a Type I error
  • Rejecting the H (^) o when it is true
• Increase the probability of making a Type II error
  • Accepting the H (^) o when it is false
• Reduce the power of the experiment to detect a true

effect as statistically significant