Nelson presentation, Slides de Estatística e Economia. Universidade Federal da Paraíba (UFPB)
thiago_moraes
thiago_moraes21 de Junho de 2015

Nelson presentation, Slides de Estatística e Economia. Universidade Federal da Paraíba (UFPB)

PDF (2 MB)
88 páginas
1000+Número de visitas
Descrição
Apresentação
20pontos
Pontos de download necessários para baixar
este documento
baixar o documento
Pré-visualização3 páginas / 88
Esta é apenas uma pré-visualização
3 shown on 88 pages
baixar o documento
Esta é apenas uma pré-visualização
3 shown on 88 pages
baixar o documento
Esta é apenas uma pré-visualização
3 shown on 88 pages
baixar o documento
Esta é apenas uma pré-visualização
3 shown on 88 pages
baixar o documento
p-value = 1

False-Positives, p-Hacking, Statistical Power, and Evidential Value

Leif D. Nelson University of California, Berkeley

Haas School of Business

Summer Institute June 2014

Who am I?

• Experimental psychologist who studies judgment and decision making. – And has interests in methodological issues

2

Who are you?

• Grad Student vs. Post-Doc vs. Faculty? • Psychology vs. Economics vs. Other? • Have you read any papers that I have written?

– Really? Which ones?

3

[not a rhetorical question]

Things I want you to get out of this

• It is quite easy to get a false-positive finding through p-hacking. (5%)

• Transparent reporting is critical to improving scientific value. (5%)

• It is (very) hard to know how to correctly power studies, but there is no such thing as overpowering. (30%)

• You can learn a lot from a few p-values. (remainder %)

4

This will be most helpful to you if you ask questions.

A discussion will be more interesting

than a lecture.

5

SLIDES ABOUT P-HACKING

6

False-Positives are Easy

• It is common practice in all sciences to report less than everything. – So people only report the good stuff. We call this

p-Hacking. – Accordingly, what we see is too “good” to be true. – We identify six ways in which people do that.

7

Six Ways to p-Hack 1. Stop collecting data once p<.05

2. Analyze many measures, but report only those with p<.05.

3. Collect and analyze many conditions, but only report those with p<.05.

4. Use covariates to get p<.05.

5. Exclude participants to get p<.05.

6. Transform the data to get p<.05.

8

OK, but does that matter very much?

• As a field we have agreed on p<.05. (i.e., a 5% false positive rate).

• If we allow p-hacking, then that false positive rate is actually 61%.

• Conclusion: p-hacking is a potential catastrophe to scientific inference.

9

P-Hacking is Solved Through Transparent Reporting

• Instead of reporting only the good stuff, just report all the stuff.

10

P-Hacking is Solved Through Transparent Reporting

• Solution 1: 1. Report sample size determination. 2. N>20 [note: I will tell you later about how this number is insanely low. Sorry. Our mistake.] 3. List all of your measures. 4. List all of your conditions. 5. If excluding, report without exclusion as well. 6. If covariates, report without.

11

P-Hacking is Solved Through Transparent Reporting

• Solution 2:

12

P-Hacking is Solved Through Transparent Reporting

• Implications: – Exploration is necessary; therefore replication is

as well. – Without p-hacking, fewer significant findings;

therefore fewer papers. – Without p-hacking, need more power; therefore

more participants.

13

SLIDES ABOUT POWER

14

Motivation • With p-hacking,

– statistical power is irrelevant, most studies work • Without p-hacking.

– take power seriously, or most studies fail • Reminder. Power analysis:

Guess effect size (d) • Set sample size (n)

• Our question: Can we make guessing d easier? • Our answer: • Power analysis is not a practical way to take

power seriously

No

How to guess d?

• Pilot

• Prior literature

• Theory/gut

Some kind words before the bashing

• Pilots: They are good for:

– Do participants get it? – Ceiling effects? – Smooth procedure?

• Kind words end here.

Pilots: useless to set sample size

• Say Pilot: n=20 – �̂� = .2 – �̂� = .5 – �̂� = .8

In words – Estimates of d have too much sampling error.

In more interesting words

– Next.

Think of it this way Say in actuality you need n=75 Run Pilot: n=20 What will Pilot say you need? • Pilot 1: “you need n=832” • Pilot 2: “you need n=53” • Pilot 3: “you need n=96” • Pilot 4: “you need n=48” • Pilot 5: “you need n=196” • Pilot 6: “you need n=10” • Pilot 7: “you need n=311”

Thanks Pilot!

n=20 is not enough. How many subjects do you need

to know how many subjects you need?

n=25

n=50

?

Need a Pilot with… n=133

n=50

n=100

?

Need a Pilot with… n=276

n

2n

?

Need: 5n

“Theorem” 1

How to guess d?

• Pilot • Existing findings • Theory/gut

comentários (0)
Até o momento nenhum comentário
Seja o primeiro a comentar!
Esta é apenas uma pré-visualização
3 shown on 88 pages
baixar o documento