item analysis notess, Summaries of Psychology

Download item analysis notess and more Summaries Psychology in PDF only on Docsity!

Item Analysis

PSYC 3200: Tests and Measurement

Learning Objectives

• Understand Item Analysis Approaches
  • Examining Descriptives
  • Item Difficulty
  • Item Discriminability
• Be able to interpret statistics from an Item Analysis
• Be familiar with Item Response Theory and Adaptive Testing

IRT Difficulty

IRT Discriminability

Item Difficulty (Ease)

• Proportion of people “passing” item
  • Number passing / total attempting
• Optimum Item Difficulty
• Ideal Range
• Relationship to instrument sensitivity

Item Difficulty p diff N Item difficulty, diff :^ =

chance

p optimum diff

= Example Optimum d, with four response options: 𝑜𝑝𝑡𝑖𝑚𝑢𝑚 𝑑𝑖𝑓𝑓 =

=. 625

Distribution of Item Difficulty [0.313, 0.384] (0.384, 0.456] (0.456, 0.528] (0.528, 0.600] (0.600, 0.672] (0.672, 0.744] (0.744, 0.816] (0.816, 0.888] 0 1 2 3 4 5 6 7

Item Difficulty Histogram

Item Discriminability

• What is it?
• Extreme Group method
• Point-Biserial Method

1

x

pbis

y x

Y Y P r s P   − =   −  

i t b

d = P − P Problems with rpbis in small tests

The Item Characteristic Curve 0

1 < 50 50-59 60-69 70-79 80-89 90- ICC For Two Questions Q 24 Q 7

Picking best items based on discriminability & difficulty

0.000 0.100 0.200 0.300 0.400 0.500 0.600 0.700 0.800 0.900 1. Difficulty Discrimination Scatter Plot for Discriminability vs. Difficulty

Several mathematical manifestations

• Most popular is the logistic approach
• Logistic models are used to predict dichotomous outcomes
  • Event = 1, Non-event = 0
• Three (+1) versions
  • 1PL Model
    • Rasch Model
  • 2PL Model
  • 3PL Model

IRT Allows for Adaptive Testing

• Instead of a fixed set of questions administered to everyone, emphasize finding the right set of questions to ask people that characterize the underlying trait.
• Uses a search algorithm to determine the difficulty level of questions participant can answer.
• Samples around that to obtain precise estimates of s.e.m.
• The emphasis is on reliably estimating the threshold level of a trait.

Varying Difficulty and Discriminability

3PL Model