Download Essay Items: Understanding the Use and Benefits of Essay Questions in Assessments and more Study notes Human Development in PDF only on Docsity!
Essay Items
Hope You Can Write
Difficulty Index ☺☺☺☺ (really pretty easy)
O
ne of the things that we will stress throughout this part of Tests & Measurement for People Who Hate Tests & Measurement is that the type of item you use as an assess- ment tool is closely tied to the level of information (or level of thinking skills) you want your test taker to show that he or she does (or does not) have. In Chapter 6, we emphasized how short answer items are best used when trying to assess memorization of factual information. In this chapter on essay items, you’ll find other tools that you can use to assess other types of outcomes. Here, we’ll focus on essay items —those items where the test taker is expected to write a coherent and informative response to a ques- tion. Forget about that Friday spelling test or even the SAT—essay tests are the real thing when it comes to seeing how well test takers integrate ideas and how well they can express them in written form.
Essay questions allow for perhaps the most unrestricted type of written assessment item that we will cover in Tests & Measurement for People Who Hate Tests & Measurement. What you want to know is how well the test taker can organize information and express his or her ideas in writing. That’s why the really, really big exams in one’s academic career are usually of the essay type—these types of items just tap more higher-level and complex skills.
WHEN WE USE ‘EM
AND WHAT THEY LOOK LIKE
Essay questions come in two basic flavors: open-ended (also called unrestricted or extended) questions and closed-ended (also called restricted) questions. An open-ended (or unrestricted response ) essay question is one where there are no restrictions on the response, including the amount of time allowed to finish, the number of pages written, or material included. Now, it is a bit impractical to allow test takers to have 25 hours to answer one essay question or to write hundreds of pages about anything they like. So, of course there are practical limits. It’s just that the limits do not define the scope of the response. For example, here’s an open-ended essay question.
- Discuss the various theories of human development that have been talked about this semester. Among other things, be sure to compare and contrast the basic assumptions of the theories, the method used to study development, and the criticisms of each. You are free to take as much time as you need. Write as many pages as you like, and organize your answer as you see fit.
That is a nice, open-ended question where the response is unrestricted. Now, take a look at this closed-ended question where the response is restricted.
- Compare and contrast two of the basic theories of human development that have been discussed this semester. Include a brief summary of each theory, the method used to study devel- opment, and the criticisms of each perspective. Limit your response to five written pages, and do not write for longer than 2 hours.
These two types of questions reflect different types of experiences. The first question, which is much less restrictive, gives the test taker a lot more flexibility (among other things) and allows for a more creative approach. The more restricted closed-ended question places definite limits on the content as well as the format.
120—— Part III ♦ The Tao and How of Testing
Essay items are the item of choice if you want an unrestricted response and want to access higher-order thinking, such as the relationship between ideas and the pros and cons of a particular argument.
THINGS TO REMEMBER: WHEN TO WRITE
- Be sure the question is complete and clear. This one sounds simple and it may indeed be, but sometimes essay questions are not very clear in their presentation. Want to know why? Because it’s not clear what the person writing the question wants to know. For example, here’s an unclear essay question
- Discuss the impact of the Civil War on the economy of the postwar South.
It’s not like this is that poorly designed a question, but it sure does not reflect a clear notion of what was learned or what is being assessed. This is the kind of a topic that some historian could write seven volumes about! Look how much more clear the following question is.
- Discuss the impact of the Civil War on the economy of the postwar South, taking into account the following factors: reduction in the work force, international considerations, and the changing role of agriculture.
122—— Part III ♦ The Tao and How of Testing
- Historical Period
- Assumptions
- Critical Elements
- Stages
- Dynamic Nature
- Structural Elements
Figure 7.1 Outline
As you can see, there are different levels of headings. By clicking various buttons on the toolbar right above the outline, the material can be rearranged, expanded, or collapsed, and many other operations can be performed. Are outlines for you? Your word processor has the option—try it and see.
- Have all test takers answer the same questions. This just reduces the burden placed on the developer of the test, but more important, makes the test easier to score—it’s just more practical. Why?
This second example just provides additional direction, which may be exactly what the test taker needs to answer the question more completely.
- Essay questions should be used only to evaluate higher-order outcomes, such as when comparisons, evaluations, analyses, and interpretations are required. Want to know what 64^3 is? (262,144)—the infant mortality rate of the United States in 2001? (6.9 per 1,000 live births)—what the French called tomatoes? (pomme d’amour for apple of love). If so, an essay question is not what you are looking for—you want the kind of item that tackles lower-level thinking skills such as knowledge or memorization. For the higher-order outcomes, such as evaluation and synthesis, you want the essay format and questions such as the following:
- Identify the principle factors that contribute to infant mor- tality in the United States and discuss and evaluate the success of efforts that have been taken to close the gap between white and nonwhite populations.
Such a question requires a knowledge of rates of infant mortal- ity but goes far beyond the statement of just facts. In fact, here’s this chapter’s cheat sheet about what kinds of words signal higher-order skills (and these words usually show up in essay items).
Chapter 7 ♦ Essay Items: Hope You Can Write 123
Analyze
Apply
Arrange
Classify
Contrast
Compare
Create
Evaluate
Explain
Generalize
Infer
Integrate
Justify
Organize
Persuade
Predict
Summarize
Synthesize
- The use of essay questions increases security because it is very dif- ficult to plagiarize during an essay item examination. And, along the same lines, it is almost impossible for test takers to effectively guess the correct answer—so guessing is removed as a legitimate concern.
- The essay item format has unparalleled flexibility. Take the time necessary to create a really good question—one that is exciting to consider and one that gives the test taker a fair chance at success.
- Essay questions can be relatively easy to construct. If you know your material well (and we would sure assume such), you can put together four essay questions in less than an hour that can effec- tively tap higher-order learning. Sure beats creating 100 multiple- choice items at 10 minutes each (yep, about 15 hours).
Why Essay Items Are Not So Good
- They emphasize writing. Well, no kidding—that’s what they are supposed to do. But what they don’t do well at all is tapping the test taker who is knowledgeable about ideas and their relationship to one another, but just cannot express it in words. For these kinds of test takers, evaluating them fairly and accurately is always a challenge.
- They can be tough to write. But didn’t we just say above that they were easy to write? Well, we said they can be. They can be a bear as well! The test designer has to invest a great deal of time in creating each essay question and making sure that it taps the objectives that are to be tested, but does so in a way to encourage the expression of ideas that you can’t find in a true/false or matching test.
Chapter 7 ♦ Essay Items: Hope You Can Write 125
Among the many reasons why essays are particularly useful is that they are very flexible in both form (the size and complexity) and purpose (relate simple ideas or elaborate a complex argument).
THINGS TO REMEMBER: WHY ESSAYS?
- Precision in sampling counts. Because essay questions take a good deal of time to create and even more time to complete as part of a test, it’s tough for the test to adequately sample the entire universe of what the test taker might have learned. This should always be a consideration in the overall evaluation of knowledge and under- standing of any topic.
- Essay questions are not easy to score. Think about it—in a class of 25 students, each of whom completes five essay questions—that’s more than 120 questions that have to be graded. No teacher will begrudge grading them (well, maybe a few), but all will have some serious problems remaining neutral, staying on task, and being consistent.
- Writing can become more important than content (see #1 above). Because essays are written, some students can bluff their way through the answer by virtue of their excellent writing ability. Just about the only way to counter this is through the use of model answers, as discussed in the next section.
HOW TO SCORE ESSAY ITEMS
Essay items are absolutely terrific and almost indispensable for sampling higher-order thinking. I’ve mentioned that several times throughout this chapter. But they are a bear to score—time con- suming and very demanding of all the scorer’s attention. With that in mind, here are a few tips that might make the scor- ing process more efficient and result in a fairer assessment.
- Scorers should provide plenty of time to score an essay item. Each of the items has to be read and then scored, and often the scorer will read the items more than once—the first time for a general overview of the content, and the second time for a more detailed analysis, including an assessment of content (again) and writing skills (such as grammar, transitions, and sentence usage).
126—— Part III ♦ The Tao and How of Testing
- If possible, grade the responses without knowing the test taker’s iden- tity. Because a subjective element can enter into the grading of essay questions, not knowing who the test taker is (and avoiding that pos- sible bias) can be a great help. There are a bunch of ways to do this (using code numbers keyed to names rather than just names is one example), but another way is to have the test taker put his or her name on the back (and blank) side of the first page of the responses. The grader won’t see it and any conflict can be avoided.
- Thesis and organization
- Content knowledge
- Writing style and mechanics
Each of these elements, for example, could be scored separately. A model too difficult for you to do? Then at least have the impor- tant points listed on a separate sheet of paper against which you can compare test takers’ responses.
- Score each question across all test takers. Given that you have a model answer, go through each test focusing the same question each time. For example, score the answer to Question 1 across all test takers, and then go back and score Question 2 across all test takers. This allows you to make absolute judgments in comparison to your model answer, but also to make relative judgments (if nec- essary) within any one item.
128—— Part III ♦ The Tao and How of Testing
Essay Items and Unreliability
You remember from Chapter 2 that our discussion about reliabil- ity focused in part on the reduction of error variance. The more error variance that is removed from the entire testing situation, the higher the reliability. OK—given that thought, there are a lot of sources of error variance when it comes to essay items, and perhaps the largest source is differences in grading. By its very nature, an essay item has so many indefinite things associated with it (objectivity of the grader, student’s writing skills, etc.) that it is even more important to control what you can. So, do try and ensure anonymity, use a model for scoring, and standardize conditions whenever possible. You’ll have a more reliable test, and the test takers will be treated more fairly.
SUMMARY
Essay questions are terrific to get at those more sophisticated thinking skills that we have to assess to find out if an individual understands more complex ideas and how these ideas relate to one another. But although they might be relatively easy to create (they are short, after all, and only a few are needed), they can underrep- resent content, they are tough to score, and there are lots of reasons why they could be considered unreliable. So, use ’em only when they need to be used!
TIME TO PRACTICE
- In your area of interest, write one stunningly terrific essay question. Then, exchange it with a classmate and evaluate it according to the guidelines that were presented in this chapter.
- Explain why essay questions are more useful for tapping higher-order thinking skills than just straight memorization.
- Name two advantages of essay items and provide an explanation as to why they are advantages.
- Write three essay items that violate at least one of the guidelines we identified in this chapter, and indicate what’s wrong.
- Name two disadvantages of essay items and discuss how you might compensate for them.
ANSWERS TO PRACTICE QUESTIONS
- Here’s my question:
Discuss the origins of the testing movement in the United States and be sure to identify how the social and political events of the times helped encourage the growth of the testing industry (45 minutes).
- Unlike test items such as short answer or completion (or many other types you have yet to learn about), essay questions provide the flexibility to explore ideas and not just require memorization or the repetition of facts.
- First, good performance on an essay question demands an understanding of ideas and how they relate to one another. That’s very important in topics where there is a premium on relationships between ideas and concepts rather than rote memo- rization of those concepts. Second, they are very flexible in that they can be adjusted to meet the needs of the assessment setting. One set of essays might be used to look at a description of a particular historical event, whereas a more
Chapter 7 ♦ Essay Items: Hope You Can Write 129
examination were compared with students with the opposite pattern (top third on the multiple-choice questions and bottom third on the essay questions). Part of the findings? Students who were strong in the essay format and weak in the multiple-choice format were as successful in their college courses as students with the opposite pattern. Guess if you know it, you know it.
And on the Internet
- The Advanced Placement people at the University of Georgia can show you some real live essay questions from previous years dating back to the early 1960s at http://apbio.biosci.uga .edu/exam/Essays/. This is a perfect place to see what essay questions look like in the area of biology at some of the most advanced levels.
- Computers grade the prose of an essay question? Maybe. Read all about it at Salon.com’s technology section, located at http:// www.salon.com/tech/feature/1999/05/25/computer_grading/.
Chapter 7 ♦ Essay Items: Hope You Can Write 131
