Automated Item Generation
André F. De Champlain, PhD
Director, Psychometrics and Assessment Services
Medical Council of Canada
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Automated Item Generation: A Fresh Approach to Creating Assessment Items, Assignments of Information Technology Management
Automated Item Generation (AIG) is a process that uses item models and computer technology to generate test items. The methodology involves identifying a problem, specifying sources of information required to diagnose it, and describing key features within each information source. Item models are created using this information, and software like IGOR is used to systematically combine it to produce new items. The Medical Council of Canada (MCC) has been working on AIG with the University of Alberta for over 5 years, generating tens of thousands of items across 50+ cognitive maps. Predictive identification accuracy ranges from 32% to 52% across experts, and AIG items are on average more difficult and discriminating.
Typology: Assignments
2021/2022
1 / 10
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Automated Item Generation: A Fresh Approach to Creating Assessment Items and more Assignments Information Technology Management in PDF only on Docsity!
Automated Item Generation
André F. De Champlain, PhD Director, Psychometrics and Assessment Services Medical Council of Canada
What is Automated Item Generation (AIG)?
- Automated item generation (AIG) is the process of using item
models to generate test items with the aid of computer technology
- AIG uses a 3-stage process for generating items where the
cognitive mechanism required to solve the items is identified and manipulated to create new items
The Item Writing World As We May Come to
Know It….
The Item Writing World As We May Come to
Know It…
AIG Methodology
- Item models are created using the cognitive model content, where an item model is like a template, a rendering, or a mold of the assessment task (i.e., it’s a target where we want to place the content for the item)
- A 54-year-old woman has a
. On post- operative day . The patient has a temperature of 38.5c. Physical examination reveal . Which one of the following is the best next step? - Type of Surgery: Gastrectomy, Right Hemicolectomy, Left Hemicolectomy, Appendectomy, Laparoscopic Cholecystectomy
- Timing of Fever: 1 to 6 days
- Physical Examination: Red and Tender Wound, Guarding and Rebound, Abdominal Tenderness, Calf Tenderness
AIG Methodology
- After the item model is specified, we combine this information systematically to produce new items
- To accomplish this complex combinatoric task, we created software for item generation called IGOR (Item GeneratOR)
- IGOR was programmed using Sun Microsystems JAVA
Next Steps for the MCC
- Undertake 2 additional AIG content development workshops in May and September, 2015 - ~ 20 new cognitive maps
- Pretest 80-100 AIG items in the spring, 2015 MCCQE Part I exam cycle
- Selected from 2014 cognitive maps and generated items
- Create several apps that will further automate the AIG process and allow us to fully transition AIG to the MCC
- Complete a cost-benefit analysis of AIG vs. traditionally written items