creating test items notes, Summaries of Psychology

Download creating test items notes and more Summaries Psychology in PDF only on Docsity!

QUESTION WRITING

PSYC 3200: Tests and Measurement

Learning Objectives

• Become familiar with good principles of question writing
• Be able to write good survey/test questions
• Understand response sets and issues with responding to survey items

Defining a Measure: Pre-pilot & Pilot

• “Shop” your questions around
  • Stakeholders
  • Content suggestions
  • Questions about meaning
• Revise again
• Pilot to small audience + Think aloud exercise
  • Ask for written feedback
  • Look for variance
  • Time respondents
• Revise again

Second Pilot

• Collect large amount of data
  • 5 to 10x number of items
• Scale Analysis (e.g., reliability and/or factor analysis)
• Item Analysis
• Validity Analysis
• Item Selection
• Cross-Validate

Phrasing Questions

• Clarity
• Level of wording
• Non specific words & phrases
• “Double-Barrelled” questions
• Manipulative Wording
• Inappropriate Emphasis
• Emotional words & phrases

Examples

• One of the few negative consequences of leaving this organization would be the scarcity of available alternatives. Meyer & Allen (1997)
• Which of the following ethnic groups do you most closely identify with? (Rea & Parker, 1997).
• Do you believe the visible development at Alpine’s freeway entrances will affect the image and property values of our whole community? (Rea & Parker, 1997).

Response Formats

• Dichotomous Scales
• Categorical (Nominal) Scales
• Likert-Type Scales
• Agreement Scales
• Value Scales
• Intensity
• Frequency
• Semantic Differential-type scales

Dichotomous True False The material presented in this class was very difficult The material presented in this class was difficult The material presented in this class was somewhat difficult

Agreement Strongly Disagree Disagree Neutral Agree Strongly Agree The material presented in this class was very difficult The material presented in this class was difficult The material presented in this class was somewhat difficult

Evaluative Very Easy Somewhat Easy Neither Easy nor Difficult Somewhat Difficult Very Difficult The material presented in this class was

Examples Cont’d. Intensity

• I liked this instructor…
  • 1. Very Much
  • 1. Somewhat
  • 1. Neither liked nor disliked
  • 1. Not very much
  • 1. Not at all Frequency
    • This instructor treated people respectfully. - 1. Never - 2. Rarely - 3. Sometimes - 4. Most of the time - 5. Always

Forced Choice Format 1 I know that I am good because everybody keeps telling me so When people compliment me I sometimes get embarrassed 2 I like to be the center of attention I prefer to blend in with the crowd 3 I think I am a special person I am no better or worse than most people From the NPI-16, Ames, Rose, & Anderson (2006)

Issues & Number of possible responses

• Categorical Scales
• Likert-Type scales (odd or even?)
  • You answered the following question:

What if…

• Only four response choices were offered?
  • Strongly Disagree
  • Disagree
  • Agree
  • Strongly Agree If we ignore the middle category, we have 25 disagreeing and 12 agreeing (25+12=37). So, 25/37=.68 or 68%, thus 68% disagreeing and 32% agreeing. Research has shown that we would expect the neutral category to split proportionally. So, of the 18 neutral, we would anticipate 68% ( people) to break to disagreement and the remainder (6 people) to break toward agreement.