Download Power Analysis Using Simulation: A Comprehensive Guide and more Study notes Innovation in PDF only on Docsity!
“I’ve Got the Power”:
How Anyone Can Do a Power Analysis
of Any Type of Study Using Simulation
Sean P. Lane Erin P. Hennes Tessa V. West
University of Missouri Purdue University New York University
San Diego, CA SPSP January 29, 2016
Part I
General Approaches to Power Analysis
Dance of the p Values
Simulation example (^) Cumming, G. (2013) Intro Statistics 9 dance of the p values
Visualizing Power
Effect Sizes vs. p Values
- Effect Size: Magnitude of effect (e.g., r, r
, β, η
, d )
- Independent of sample size
- Mean difference or association relative to variability
- p Value: Probability of obtaining an effect at least as large as what was actually observed if the null hypothesis is true
- Highly dependent on sample size
- Mean difference or association relative to the ratio of variability to sample size GOAL: Identify minimum sample to have a high likelihood (e.g., 80%) of a low likelihood (e.g., 5%) of obtaining predicted effect if null is true
Why Should I Do a Power Analysis?
- Limited Resources
- Studies are effortful, expensive, and time-consuming
- Power analyses are free and less time consuming
- Efficiency
- Increases chances of finding significant effects
- Increases chances of replicating prior effects
- Decreases null effects (file drawer)
- Increase confidence in null effects
Counterarguments
- If the effect is significant with a small sample, it must be real
- AKA, why do we care about power in the “replicability crisis” if power is about Type II errors, not Type I?
- Why don’t we just recruit a ton of people in all of our studies?
- Why don’t we just get rid of null hypothesis significance testing?
- Doesn’t power analysis take a lot of time?
What Information Do I Need?
- Generally working within a regression framework
- Effect sizes and additional sources of variability
- Using standardized effects will often reduce the number of parameters that we need to determine
- Ex. Simple linear regression
Example: Calculating A Priori Power By Hand
for Simple Linear Regression
𝑁 = 𝜎 𝜀 2 𝑧𝛼 2 − 𝑧 𝛽 2 𝛽 2 𝜎 𝑋 2 = 1 − 𝛽 2 𝑧𝛼 2 − 𝑧 𝛽 2 𝛽 2 𝜎 𝑋 2 = 1 −. 15 2 × 1. 96 − −. 84 2
. 15 2 × 1 2 = 341
Example: Calculating A Priori Power Using
G*Power for Simple Linear Regression
t tests - Linear bivariate regression: One group, size of slope Analysis: A priori: Compute required sample size Input: Tail(s) = Two Slope H1 = 0. α err prob = 0. Power (1-β err prob) = 0. Slope H0 = 0 Std dev σ_x = 1 Std dev σ_y = 1 Output: Noncentrality parameter δ = 2. Critical t = 1. Df = 341 Total sample size = 343 Actual power = 0.
Early Work
1990s: Closed form or iterative solutions for estimating standard errors and power for two-level models
- Approximation for estimating standard errors for two-level models (Snijders & Bosker, 1993; Liu & Liang, 1997)
- Accommodating CS, AR1, and TOEP residual covariance matrices (Hedeker, Gibbons, & Waternaux, 1999)
- Software for basic nested models [Optimal Design] (Raudenbush & Liu,
Early Work - Limitations
- Overly restrictive assumptions
- Approximations that had very limited generalizability
- Authors conducted analyses themselves and presented tables of sample sizes - Single papers often consisted of tables for only one type of model - Did not instruct researchers to conduct power analyses on their own
- Published in statistics and education journals
Recent Work
2000s-: More flexible simulation approaches, additional software packages
Ahn, C., Heo, M., & Zhang, S. (2015). Sample size calculations for clustered and longitudinal
outcomes in clinical research. Boca Raton, FL: CRC Press.
Bolger, N., & Laurenceau, J-P. (2013). Intensive longitudinal methods: An Introduction to diary and
experience sampling research. New York: Guilford.
Cools, W., Van den Noortgate, W., & Onghena, P. (2008). ML-DEs: A program for designing efficient
multilevel studies. Behavior Research Methods, 40, 236 - 249.
Gelman, A., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models.
Cambridge: Cambridge University Press.
Moerbeek, M., & Teerenstra, S. (2016). Power analysis of trials with multilevel data. Boca Raton, FL:
CRC Press.
Zhang, Z., & Wang, L. (2009). Statistical power analysis for growth curve models using SAS. Behavior
Research Methods, 41, 1083 - 1094.
Recent Work - Limitations
Design GPower Power & Precision PS PASS RMASS PinT Optimal Design ML-DEs Simulation* t-tests Yes Yes Yes Yes No No No No Yes Correlation Yes Yes Yes Yes No No No No Yes ANOVA Yes Yes Yes* Yes* No No No No Yes** Multiple regression Yes Yes Yes* Yes* No No No No Yes** Multilevel regression No No No No Yes Yes Yes* Yes* Yes** Path analysis No No No No No No No No Yes Latent variables No No No No No No No No Yes
