Partial preview of the text
Download Preparing effective instructions for test tasks and more Study notes Teaching method in PDF only on Docsity!
Language Testing in Practice: Designing and Developing Useful Language Tests Lyle F. Bachman and Adrian &. Palmer Oxford University Press 1996 10 Preparing effective instructions Introduction As discussed in Chapter 3, the way test takers perform on language tests is affected to some extent by the characteristics of the tests themselves. In keeping with the maxim that we want to make it possible for test takers to do their best, ir is essential thar they clearly understand how they are to proceed in taking the test, the cypes of tasks they are going to encounter, the ways in which they are cxpected to respond to these tasks, and how their responses are going to be cvaluated. ‘he test instructions are pacticu- larly imporvant because it is through them that we inform test takers how they are expected to approach aid attempe the test tasks. The instructions will reflect the considerations that we have included in the design of the test (discussed in Chapters 5-8) and will thus communic- ate to the test takers our intentions as test developers. These include matters such as the purpose for which the test is being used, the language abilities to be tested, and the relative importance of different parts of the rest. We convey these intentions explicitly by what we include in and the way we present the instructions. In this chapter we will discuss ways in which instructions can be most effectively specified to promote the best test per- formance in a given group of test takers. We will discuss general principles and what we believe are the essential corsponents of instructions, along vith suggestions on how these can be effectively written and presented t0 test takers. Purpose of instructions The instructions are typically the first part of the test that test takers encounter. It is the instructions, therefore, that bear much of the responsib- ility for setting the test takers’ expectations and appropriately motivating them to do their best on the test. The primary) purpose of the test instruc- tions is thus to assure that the test takers understand the exact nature of the testing procedure and of the test tasks, how they are to respond to these tasks, and how their responses will be evaluated. The essential com- ponents of the test instructions, therefore, ar Preparing effective instructions 183 Channel of presentation Understandability can also be facilitated by presenting the instructions in a channel that is most appropriate to the purpose and abilities being tested, and that test takers are most likely to find easy. In tests whose input is presented in the visual channel (written), instructions are also typically presented in this way. Iastructions for completing a test with tacks that require reading, for example, would typically be presented in writing, as would instructions for composition tests. However, in tests with listening or speaking activities, which typically involve input in the aural channel, instructions would also be presented in this channel. Thus, if the test were presenred ‘live’ by the person administering the test, the instructions would be read aloud to the test takers, and if the test were presented via an audio or video rape player, the instructions would be presented on the tape. Some test takers may be able to perform one type of language use activity better than another, in which case it may be helpful if instructions are presented both orally and in writing. 1 students are relatively good at ‘reading activities and weak at listening activities; for example, test takers could be given a set of written instruc tions for the tasks that involve listening and asked to follow these as they are presented orally, either by the test administrator or on a tape player. If, on the other hand, they are more able at listening tasks than. at reading tasks, the test administrator might read the written instruc tions aloud while test cakers read them silently. The purpose of pre- senting the instructions beth orally and in writing is to insure that students understand. However it is possible that some students will find this distracting, and that ix inverferes with comprehension of the imstruc tions. Since different test takers vary as to whether both oral and written instructions will facilitate or interfere with understanding, the test cevel- oper must: base the decision on 2 thorough understanding of the personal characteristics of the particular test takers, and, if possible, on the feed- back obtained from. pre-testing. Providing example tasks Providing exemple tasks can also facilitate test taking, particularly when test takers may not be familiar with the specific task type, or with complex question types. Consider the following test task, for example: Instructions: Make the necessary changes and additions to the following sets of words and phrases to produce a complete sentence for each. Write the sentence in the space provided. Tost finish/six-week data processing course/local college. 184 Language test development Even with fairly detailed instructions, it may not be entirely clear to all test takers just what their response is supposed to be. An example task such as the following could help: Example: 1 be quite happy/ceceive/letter/you yesterday. Answer: 1 was quite happy to receive a letter from you yesterday. While we generally recommend providing examples as a cans for insuring that test takers understand test tasks, there are two types of costs involved. First, providing good examples can be as difficult as writing good test tasks, Second, reading through examples requires additional time on the part of the test takers. However, if the test developer feels that a new type of test task is particularly useful, or if che ability cannot be easily tested in another more familiar way, example tasks weed to be provided. The burden of writ- ing example tasks can be alleviated to some degree by using test tasks that have been discarded after pre-testing as too easy for the test. This has the added benefit that the example tasks will be easier and hence more under- standable than the tasks in the test, Components of imstructions Many language tests ace aimed at measuring more than one aspect of language ability, or employ more than one method of testing, and thus include more than one part, With such tests, it is useful to provide, at the beginning of the test, a set of general instructions that apply to all parts of the test. In addition, if the different parts include different task types, require different procedures, or use different criteria for coz- reemess, then these need to he made explicit in specific instructions for cach part. Thus, tests with multiple parts may include both general and specific instructions. In tests with only one part, on the other hand, there need be only one set of instructions. In either case, instructions need to include descriptions of the following: Test purpose Language abilities to be tested Parts of the test and their relative importance Procedures to be followed for all parts of the test Scoring method Test purpose The purpose of the test is its intended use, and we have discussed a variety of uses in Chapter 6. How will the information obtained from the test be used? What inferences or decisions are to be made on the basis of the test scores? These will all be specified during the test development process, as 186 Language test development Language abilities to be tested The rationale for informing students of the language abilities we intend to measure is essentially the same as that for informing them of the test’s purpose. In addition, providing a statement of the abilities to be measured gives test takers the means for assessing the relevance of these abilitics 1 to the test’s intended purpose, 2 toa particular course of instruction, either already completed or about to be entered, 3. to their TLU tasks, and 4 to the types of test tasks used, As with stating the purpose of the test, we believe that an explicit statement of the abilities to be measured will help motivate students to do their best and will also help insure accountability of test use. In tnulti-par tests that measure different aspects of language ability, state- ments of the abilities to be tested should be given in the specific instruc. tions to the different parts. Since very few individuals who take language tests are either language teachers or trained linguists, it is important that our descriptions of the language abilities to be tested should be stated in non-technical language. In classroom testing, a ‘label-—a word or phrase—may be sufficiently clear for tests that are directly related to learning objectives and activit- ies. In large-scale testing, however, where test takers Tay come from a wide variety of learning backgrounds, labels are seldom sufficient, in that even an apparently obvious label, such as ‘reading comprehension’, for example, may have different connotations for different test takers and test users. For such situations we would recommend that the instruc- tions include more than a label, and that a brief description of the areas of language ability being measured should be provided. Since terms such as ‘proficiency’, ‘competence’, ‘comprehension’, and ‘communicate’ are ambiguous even to language teachers, it is not likely that they will be clear to test takers. We would therefore recornmend that the abilities to be measured should be described in terms of specific language use activit- ies, since it is these thar are more likely to be understood by test takers. Thus, rather than a statement such as ‘This test is a measure of your listening comprehension’, for example, we would prefer a statement such as “This is a test of how well you can understand spoken English in lectures and classroom discussions.’ Parts of the test and their relative importance ‘With tests that consist of several different Parts, it is important for test takers to understand Preparing effective instructions 187 an how many parts there are, 2. how many tasks there are in each part, the relative importance of each past, and task, often stated in terms of a maximum score per part or task, and 4 how much time will be allocated to each, @ ‘The general instructions should thus include descriptions of the different patts of the test, including the types and numbers of tasks in each, and how Tnuch time test takers will be given to complete each part. Depending oa the number of different parts and how distinct they are, it may also be helpful to provide a summary table, as in Table 10-1. Part Task type Number of tasks Time allowed | Listening lasks Short answer 20 10 mins questions II Reading tasks Reading passage wo passages; 20 ming followed by short 18 questions answer questions IW Writing tasks Prompt requiring an Two prompts 80 mins extended written answer Total 60 mins Table 10.1: Example description of test parts Different types of test tasks may be appropriate for different areas of language ability for a given test purpose ot geoup of test takers, and these tasks may vary in their efficiency in terms of adequately measuring the specific area in a given amount of time. For this reason there is not necessar- ily a one-to-one relationship between the relative importance of the parts of a given test, the jumber of tasks in the different parts, or the amounts of time allowed for the differnt parts. Test takers thus may not be able to determine accurately the relative importance of the different parts from either the number of tasks or the times allowed, and we therefore believe it is important to provide an explicit statement of the relative importance of each parr, fa the example above, although the listening and reading pacts are equally important (30 per cent cach), there are more tasks in the former. This may reflecc the test developer’s perception that reading tasks are rela- tively more difficult for the intended test takers. Furthermore, although there are more listening than reading tasks, more time is allowed for the latter. This difference may also reflect the test developer’s perception of the difference in difficulty. The test developer may also know that the test takers for whom this test is intended typically cequixe more time for reading tasks than for listening tasks. Preparing effective instructions 189 In other types of tests, in which either the input or the response or both ace in the audio channel, the order in which test takers answer questions and indicate their responses may be controlled as part of the administra- tion. In a test with tape-mediated speaking tasks, for example, in which rest takers listen to aural input from a tape player and record theit spoken responses on a tape recorder, the order in which they answer the different parts or tasks is controlled by the tape that presents the input. Similarly, the type and length of each spoken response will be controlled by the type of tasks posed and amount of time allowed om the cape. Scoring method In order for test takers to understand what they are expected to do, and hence to perform at their best, they need to know how their responses are going to be evaluated. If the test includes séveral parts that will be scored in the same way, the criteria for correctness can be stated in the general instructions. If different parts will be scored differently, then these criteria should be given in the specific instructions for each part, For selected responses, there are generally two considerations that need to be made explicit. First, the test taker needs to understand whether the jtem is a ‘correct answer’ type (there is only one correct answer and this is provided among the choices) or a ‘best answer" type (there may be many possible answers, and the test taker must choose the best one from among those provided in the choices). Second, will the test taker’s score be cor rected for guessing? Although corrections for guessing ave not widely used in language tests, some test takers may come from countries or educational systems-in which this is routine for multiple-choice tests. Such individuals may be reluctant to guess on the basis of partial knowledge (i.e. they can tule out one or two incorrect choices), and thus may perform differently from other test takers who are willing to do so. (These considerations are discussed more extensively in Chapter 11.) For tasks that require a limited or extended production response, the test takers’ understanding of the criteria for correctness may affect the way they approach the given test task and hence the way they perform. For example, test takers are likely to approach a writing task differently if they believe that the primary criterion for correctness is grammatical accuracy than if they know that it will be scored primarily on its content. Since test takers are likely to have preconceptions about whar criteria of correctness will be applied to their answers, and since these are likely to influence the way in which they construct their responses, we believe that it-is crucial for the instructions to state explicitly what criteria will be nsed in evaluation. 190 Language test development How extensive should instructions be? From the discussion so far, it may appear that instructions need to be long, complex, and extremely detailed. This need not be the case. Efficient, effect- ive test instructions have three qualities: 1 they are simple enough for test takers to understand, 2 they are short enough not to take up too much of the test administration time, and 3. they are sufficiently detailed for test takers to know exactly what they are expected to do. The best basis for writing understandable instructions and for knowing how much of the test administration time they will take is 2 thorough knowledge of the personal characteristics of the test takers for whom the test is intended, combined with actual try-outs of the instructions with test takers. For classroom tests, in addition to trialing the instructions, a very effective way to insure that test instructions are appropriate for the students is to ask them to write the test instructions as a learning activity. The amount of detail required in the instructions will depend on two factors: 1 how familiar test takers are with the test tasks, and 2 the number and variety of task types used in the test. Tests that consist of multiple parts and employ a variety of relatively un- familiar test tasks are likely to require complex instructions. Thus, while there may be legitimate reasons for using a variety of task types in a given. test, such as promoting authenticity and positive impact on instruction, there is no particular virtue in proliferating different task types simply for the sake of variety or novelty. Summary in order for test takers to have the opportunity to perform. at their best, it is essential that they be provided with clear instructions. The purpose of these is to insure that the test takcrs understand the exact nature of the testing procedure and of the test tasks, how they are to respond to these tasks, and how their responses will be evaluated. Effective instructions help assure test takers that the rest is relevant, appropriate, and fair. The essen- tial components of the test instructions ar 1 a statement of the purpose(s) for which the test is intended, 2 a statement of the language abilities that the test is intended to measure, 3 a specification of the procedures and tasks, and 4 a specification of the criteria for correctness. 
