Download Perspectives on Using Example Solutions in Physics Courses: Instructors and TAs and more Study notes Physics in PDF only on Docsity!
Teaching assistants’ beliefs regarding example solutions in introductory physics
Shih-Yin Lin Department of Physics and Astronomy, University of Pittsburgh, Pittsburgh, PA 15213, USA
Charles Henderson and William Mamudi Department of Physics and Mallinson Institute for Science Education, Western Michigan University, Kalamazoo, MI 49008, USA
Chandralekha Singh Department of Physics and Astronomy, University of Pittsburgh, Pittsburgh, PA 15213, USA
Edit Yerushalmi Department of Science Teaching, Weizmann Institute of Science, Rehovot, 76100 Israel
As part of a larger study to understand instructors' considerations regarding the learning and teaching of problem solving in an introductory physics course, we investigated beliefs of first-year graduate teaching assistants (TAs) regarding the use of example solutions in introductory physics. In particular, we examine how the goal of promoting expert-like problem solving is manifested in the considerations of graduate TAs choices of example solutions. Twenty-four first-year graduate TAs were asked to discuss their goals for presenting example solutions to students. They were also provided with different example solutions and asked to discuss their preferences for prominent solution features. TAs’ awareness, preferences and actual practices related to solution features were examined in light of recommendations from the literature for the modeling of expert-like problem solving approaches. The study concludes that the goal of helping students develop an expert-like problem solving approach underlies many TAs’ considerations for the use of example solutions. TAs, however, do not notice and do not use many features described in the research literature as supportive of this goal. A possible explanation for this gap between their belief and practices is that these features conflict with another powerful set of values concerned with keeping students engaged, setting adequate standards, as well as pragmatic considerations such as time requirements and the assignment of grades.
PACS number(s): 01.40.gb
I. INTRODUCTION
Helping students develop an expert-like problem solving approach is an important instructional goal for the introductory physics courses, valued both by physics instructors[1- 3 ], as well as stake holders in academics and industry[ 4 ]. Indeed, instructional strategies have been developed and shown to help students improve their problem solving skills [5-15]. For example, the research comparing novice and expert problem solvers has identified expert-like approaches to problem solving involving elements such as initial problem analysis, planning ahead the solution, and ongoing evaluation of the progress made in the solution accompanied by refinement of the solver’s understanding of the principles and concepts involved[16-20]. Researchers have proposed a prescriptive problem-solving strategy [ 9 , 13 , 21 ] as an
instructional tool to explicate to some extent an expert-like approach to problem solving. This strategy includes three major components – (1) distinct problem analysis, (2) solution construction that makes explicit the plan, in particular intermediate goals and the principles used to figure them, and (3) checking the final answer. Research indicates that when instruction follows a cognitive apprenticeship approach (i.e. instructors explicitly model a prescriptive problem-solving strategy, require students to follow it, and coach them when doing so), students are likely to adopt more expert-like problem-solving approaches [ 5 ,7-12]. Although the effectiveness of these instructional strategies and relevant curricula materials has been documented in the research literature, such strategies and techniques are rarely put into common practice as intended[ 22 ]. Previous research on instructors’ beliefs [18] that influence their choice of instructional strategies and material for introductory physics courses indicates that faculty tend to value, but not use aspects of these curricula. For example, although faculty generally held the learning goal of helping students develop expert-like problem-solving approaches and were aware of problem features that could support this goal (i.e. rich context, compound problems that are not broken into parts, etc.), they refrained from integrating these features in the problems they used during instruction because they believe these features conflict with a powerful set of concerns regarding clarity of presentation and minimizing student stress [ 2 ]. Such ambivalence could have served as a fruitful lever for reflective professional development processes. However, many faculty might not be open to engage in long-term professional development. At many institutions, in addition to faculty members, teaching assistants (TAs) also play important instructional roles in the teaching of problem solving. For example, TAs are often the ones to lead recitations in which they present students with example solutions for physics problems, guide students in solving problems and assess students' solutions. A growing number of institutions require their teaching assistant to attend a training course and mentor them while they take their first steps as teachers [23-28]. Since teachers commonly construct their instructional beliefs and habits in their first years as teachers [ 29 ], the beginning of a TAs teaching career is likely a formative period that will influence later performance. Moreover, since most physics faculty have also been TAs, we expect it to be a formative stage where the beliefs of faculty develop. Thus, TA training courses provide a window of opportunity to significantly impact university physics instruction. To design effective professional development programs, it is important to know about TAs’ beliefs when entering their job. However, the research on TAs beliefs and practices in this regard is currently limited. This paper intends to fill in this gap by examining how the goal of promoting expert-like problem solving is manifested in the considerations of first-year TA choices of example solutions. (Please note, hereafter when referring to the TAs that we study, the term TA refers to the first-year TAs that we studied.) This instructional context was chosen for two reasons. First, it corresponds to one of the major responsibilities that TAs in many institutions are expected to fulfill (i.e. to guide students in problem solving and to present or demonstrate problem solutions in introductory physics courses). Second, in many of the aforementioned research-based instructional approaches [5-8, 30 ] example solutions serve to explicate the problem solving processes of an expert. In this study, we will examine the extent to which TAs value modeling expert-like problem solving and how, if at all, they realize this goal through the design features of example solutions they would present to their students. For the purpose of our study, we use Reif’s prescriptive problem solving method [ 9 , 13 , 21 ] as the standard against which we measure the features.
Research indicates that when instructors explicitly model and encourage students to follow a set of problem solving strategies explicating expert-like behavior, students are likely to adapt a more expert-like problem-solving approach [ 5 ,7-12]. These strategies are also referred to in the literature [ 9 ] as prescriptive problem solving strategies. One should note that these prescriptive strategies reflect only some aspects of the actual problem solving process that an expert goes through and leaves out other aspects, such as back and force moves where the solver reconsider their approach to the solution. Several instructional techniques that have been used to promote expert-like problem solving involve “real” problems [ 5 , 6 , 14 , 15 ] that require a higher level of analysis and planning to encourage students to adopt more expert-like approach. They also involve modeling, coaching, and fading phases according to the cognitive apprenticeship framework [ 38 ]. Based on this framework, in addition to introducing problem-solving strategies that reflects the implicit problem-solving approaches used by experts, instructors should also demonstrate use of the strategies during class [ 9 , 13 , 21 ]. In particular, example problem solutions that instructors present to their students can be used to make explicit the prescriptive problem-solving strategies. Then, the students can be provided with opportunity to practice applying the strategies (e.g., by working with other students or with computers) under the assistance of the instructor. Scaffolding can be provided in one-on-one tutoring, by cooperative learning groups [ 7 ] and by computer-based tutoring [ 8 ]. As students develop more independence and expertise, the support from the instructor can be gradually reduced. In the following section, we briefly summarize the research related to how example solutions can be structured to promote learning.
B. Structuring example solutions to promote learning
In this paper, we use the term “example solutions” broadly to refer to any problem solutions that students are exposed to during an introductory physics class. This includes solutions that the instructor works on the board during lecture and recitation, written solutions in the textbook or other similar material, and written solutions that the instructor distributes to students after students have submitted solutions to homework or test problems. Example solutions are used in nearly all introductory physics courses. They are described by a number of different names in the research literature, such as worked examples [39-42], worked-out examples [43-46], instructor solutions [ 47 ] and example problem solutions [ 1 , 3 , 48 ]. As mentioned earlier, research suggests that modeling the tacit problem-solving approaches of experts is an important instructional role of example problem solutions. However, there is a difference between how good students and poor students study example solutions [ 46 , 49 ]. Students who “self-explain” the example solution more are more likely to benefit more from reading the example solution. Self-explanations refer to the content-relevant articulations (beyond what the text explicitly said) formulated by a student in order to make sense of an example solution [ 50 , 51 ]. This commonly involves filling in the gaps that correspond to the omissions in the solution and/or resolving the conflict between the students’ mental model and the solution[ 51 ]. Chi [ 51 ] argued that high self-explainers are those who readily detect such conflicts while learning from an example solution. It is recommended that instructors provide students with prompts that encourage students to detect conflicts. Atkinson, Renkl, and Merrill [ 45 ] have shown that principle-based prompts
are effective in inducing principle based self-explanations, one style of the self-explanation that successful learners use [ 43 ]. Research on the design of example solutions has shown that example solutions are more effective if multiple sources of information (e.g., diagram, text, and aural information) are integrated into a unified presentation [ 39 ,52-54] to avoid splitting students’ attention across multiple non-integrated sources of information, which may cause cognitive overload on students [ 52 , 53 ]. In addition, structuring the examples to emphasize the important chunks of steps or subgoals (either by explicitly labeling them or simply isolating them visually) can guide students to discover the underlying deep structure of the solution and enhance learning [ 39 ,55-57]. With the aid of these structural cues, students are encouraged to explain to themselves why the steps are chunked together, which can promote the induction of generalization. It is found that, compared to students who learned from traditionally formatted examples solutions, students who learned from solutions in which subgoals were highlighted were more likely to transfer what they learned correctly into a new context [ 55 ]. Research also indicates that at the initial stages of skill acquisition, learning from example solutions is effective for improving problem solving performance compared to problem solving itself [ 39 , 58 ]. Because the cognitive overload is less when studying example solutions than actually solving problems, more short term memory capacity is available for students to extract useful strategies and to develop knowledge schemas [ 39 , 58 , 59 ]. At this stage, process-oriented solutions (solutions which present the rationale behind solution steps) are appropriate [ 41 ]. On the other hand, as learners acquire more expertise, process-oriented examples become less effective (or in some cases may even start to hamper learning) because the redundant information presented, which is hard to ignore, takes up unnecessarily large amounts of the limited working memory [ 41 , 60 ]. Product-oriented solutions (in which rationale is not included) are therefore more appropriate at the later stage of learning when the learners possess more prior knowledge. Although educational researchers have provided insights into how the example solutions may be used to promote student learning, whether or not these research findings are incorporated into real practice by an instructor depends on the instructor’s familiarity with the research findings as well as their beliefs about teaching and learning. In the following sections, we briefly summarize the research on instructors’ (faculty and TAs) beliefs and practices.
C. Faculty Beliefs and Practices
A number of researchers have investigated the general ideas of teaching and learning held by college instructors [61-67]. These studies generally found that instructors’ views could be characterized on a continuum from teacher-centered, which emphasizes the transmission of knowledge, to student-centered, which emphasizes the construction of knowledge by the students. Instructors’ beliefs about problem solving has also been found to be correlated with their general ideas about teaching and learning [ 66 ]. For example, instructors who conceived the meaning of the problem as obvious or unproblematic to students and thought of problem solving as an application of existing knowledge held more teacher-centered views of teaching. On the other hand, instructors who see that the meaning of the problem is not necessarily obvious to students and that problem solving involves making sense of the problem held more student-centered views [ 66 ].
building physics knowledge may disregard students’ common sense idea, which contradicts the intention of the tutorial design [ 24 ]. Since most TAs are graduate students a general understanding of graduate students’ attitudes to problem solving in introductory physics provides another perspective on TAs’ beliefs and practices. Mason and Singh [ 75 ] compare the attitudes and approaches to problem solving in the introductory physics courses by graduate students vs. introductory students. While close to 90% of the graduate students reported that they explicitly think about the underlying concepts when solving introductory physics problems, more than 30% of them perceived problem solving in introductory physics as essentially plug and chug. An examination of the graduate students’ written explanations suggests that this view results from their relative expertise in solving introductory physics problems. Many of them can immediately realize the principle that should be used when solving a problem, therefore, perceiving the task as one that does not require much thought and reflection. Indeed, some graduate students noted that they do not need to reflect and learn from the problem solutions after problem solving in introductory physics because the problems are obvious to them and they feel that reflection is not needed. Several institutions conduct TA and LA training programs [23-28]. A hallmark of successful training programs is matching the program to TA strengths and capabilities [ 27 ]. One of the goals of this study is to inform professional development providers to help them better support TAs to become more successful teachers.
III. METHODOLOGY
A. Participants
Twenty-four first-year physics graduate students from the University of Pittsburgh were involved in this study. All 24 TAs were enrolled in a semester-long training course led by one of the authors (CS)i, aimed at preparing them for their TA jobs. Most of the TAs in our study were simultaneously doing their TA job: 15 of them were working with the introductory algebra- or calculus-based physics course either leading recitations, lab sections, or being a grader, 2 of them were teaching astrophysics, and 1 of them was teaching an optical lab. The few TAs who were enrolled in the course and did not simultaneously have any teaching responsibilities were expecting to teach in the near future. In addition to teaching, these TAs also helped as tutors in the physics exploration center where introductory physics students could come for help in solving homework problems. Although many of the TAs were simultaneously doing their TA jobs, the study was conducted at the beginning of the semester when the TAs had just entered the graduate school. Therefore, it portrays the TAs’ preliminary ideas when they had not have extensive experience in teaching introductory physics. Thirteen of the TAs involved in this study were international students, most of whom had their former undergraduate education in China or India.
B. Methodological consideration on the study of TA beliefs
Different methods have been used (and usually combined) by researchers to learn about instructors’ conceptions and beliefs. For example, in addition to open-ended questions on a questionnaire or in an interview, researchers have combined open-ended questions with
classroom observations [ 76 ] or descriptions of concrete hypothetical teaching situations [ 77 , 78 ]. Although instructors’ teaching practices are influenced by their beliefs, simply observing the classroom practice may not reveal a complete picture because the actual practice may be a resolution of different beliefs (which may sometimes conflict each other) as well as situational constraints. Studies have also shown that similar teaching behaviors can be supported by different beliefs [ 72 , 73 , 79 ]. Thus it is difficult to infer beliefs based solely on observations of practice. On the other hand, the research literature also indicates that simply asking instructors (e.g., in an interview) about their conceptions is frequently not fruitful because conceptions can be implicitly held [80-83]. Interviews designed around concrete instructional settings have been shown to be successful in eliciting context-specific conceptions that may not otherwise be evident to the person who holds them [ 1 , 3 , 77 , 78 ]. This study introduces a methodology that is a variation of interviews designed around concrete instructional settings. We developed this methodology, called the Group Administered Interactive Questionnaire (GAIQ), to accomplish the following research goals [ 84 ]: Encourage TAs' introspection and articulation about their beliefs regarding the design of example solutions Triangulate findings regarding the above in various contexts, more and less concrete, more and less close to the actual decision-making Minimize distortion of data collection by researchers' personal bias (reliability) Compare results with pedagogies based on educational research
The GAIQ methodology represents a variation of interviews designed around concrete instructional settings that was used by two authors (Henderson and Yerushalmi) in a previous study of the considerations that shape faculty instructional choices regarding example solutions [ 1 , 3 ]. In the previous study, the data collection tool made use of semi-structured individual interviews that involved 2 types of questions: general questions about how and why the instructors might present solution to students, and concrete questions asking instructors to compare and make judgments about a set of artifacts that differed in features reflecting instructional approaches discussed in the research literature [ 1 ]. In the GAIQ methodology, the artifacts and questions asked to participants were similar to those used in the semi-structured individual interviews previously conducted. These artifacts and pre- determined questions serve to standardize data collection in order to collect reproducible data in both the interview and the GAIQ. However, in the GAIQ methodology, the individual interview was replaced with a written questionnaire (as shown in FIG. 5, FIG. 6 and FIG. 7) to respond to several concerns. First, from the practical perspective, individual interviews require significant time for both data collection and analysis. The GAIQ approach takes advantage of the TA training course to streamline data collection. Second, within semi- structured individual interviews, the interviewer interventions required to clarify respondents' answers may affect the reliability. In the GAIQ tool, clarification takes place via a sequence of worksheets, thus standardizing researchers' interventions. Finally, as the data collected in individual interviews is extremely rich, there is ambiguity in categorization of the data. The GAIQ approach incorporates elements intended to have the respondents clarify meaning (therefore achieving the validity of data suggested by Kyale [ 85 ]) by allowing the respondents to share and articulate their ideas in a group discussion as well as having them categorize some of the data themselves. The GAIQ methodology is described in more detail
problem solutions. Even if the TAs may not have extensive experience in teaching introductory physics, we do believes that the TAs would be familiar with the example solutions provided because these solutions were designed to reflect the actual practice in common physics classroom. In these regards, we expect the TAs to perceive the activities as relevant and reasonable. We made clear in the instruction that TAs should think as instructors when writing their own solutions and filling out the worksheets (Please refer to figures 5 and 6 for example.)
D. Data collection procedure
Table 1 summarizes the GAIQ data collection sequence. In the pre-lesson stage, as part of their homework, TAs were asked to write a solution (to the problem presented in FIG. 1) that they would hand out to their students. TAs were then asked to respond to open-ended questions presented in FIG. 5 regarding how they think example solutions should be used in their instruction. The TAs were also provided with three example solutions for the problem (shown in FIG. 2, FIG. 3, and FIG. 4) and were asked to fill in a pre-discussion individual worksheet (FIG. 6) where they identified prominent design features of the solutions, ranked the solutions based on i) which solution has more of each feature and ii) their preference for including each design feature in solutions, and explained their reasons. In the lesson stage the TAs interacted in small groups to share their ideas with peers, followed by a whole-class discussion. The lesson stage represented about 120 minutes of class time. In addition to the author who led the training course (CS), another author (SL) also sat in the classroom to take field notes. Finally, in the post-lesson stage the TAs were provided with the opportunity to explain whether (and why) their preference changed by filling in a post-discussion worksheet (FIG. 7 ). On this post-discussion worksheet they were also asked to match the features they identified on the pre-discussion worksheet to a list of pre-defined features (presented in Table 2 ; TAs were only given the descriptions of each feature and not the information about clusters) corresponding to different aspects of the solution presentation. The list represents features identified in a pilot study with a similar group of first-year graduate students, who were enrolled in the same TA training course in the previous year and were asked to discuss the prominent features they observed in the same set of solutions artifacts on a worksheet similar to the pre-discussion worksheet used in our current study. The list of pre-defined features was created by coding the TA responses in the pilot study. With the help of this list of pre-defined features, the post-discussion worksheet allows the TAs in our current study to participate in the categorization of data, which promotes the validity of the study. The features in Table 2 can be grouped into clusters to help in the interpretation of data. The first three clusters (C1 to C3) relate to the key stages in a prescriptive problem solving model described in the literature [ 13 ]. The final two clusters (C4 and C5) relate to communicating the solution to students. Each is briefly described in the following paragraphs. The first cluster (C1) relates to the initial problem analysis as described by Reif [ 13 ]: "The purpose of the initial problem analysis is to bring the problem into a form facilitating its subsequent solution. To this end, one must first clearly specify the problem by describing the situation (with the aid of diagrams and useful symbols) and by summarizing the problem goals." (p. 27)
The second cluster (C2) relates to the search process that is the core of the solution construction stage described by Reif. The problem decomposition that takes place in this stage is materialized via feature 4 (explicit sub-problems are identified). Feature 6 (principles/concepts used are explicitly written) makes explicit the relations used in each sub- problem to eliminate unknowns. The rationale underlying the problem decomposition is described in feature 3 (providing a separate overview) and feature 5 (reasoning is explained in explicit words). A specific solution cannot demonstrate the recursive nature of the search process, yet feature 10 (providing alternative approach) reminds us that there are alternative solution paths. Finally, feature 12 (forward vs. backward solution) reflects possible directions of the search process. The third cluster (C3) relates to checking the solution. For example, a symbolic solution (F13) allows one to check that the different stages in the solution are self-consistent. Performing a check of the final result by examining the unit or the limiting cases (F14) allows one to contemplate whether the final answer makes sense. The fourth cluster (C4) and the fifth cluster (C5) are both related to the presentation of the solution. Features in the C4 clusters are all related to the “long/detailed” aspect of a solution. F7 (thorough derivation) and F9 (details without which the solution is still technically correct) represents two example components in a solution content that can lead to a long physical length (F8). On the other hand, the single feature in cluster C5 focuses whether the solution is presented in a clear and organized way (F11).
E. Data analysis
The pre- and post-discussion worksheets as well as TAs’ own solutions and their responses to the open-ended questions were collected for analysis. To answer research question (1a), TAs’ goals were identified by analyzing (i) their answers to the open-ended questions asking them to describe the situations and purposes for providing example solutions to students in a general context and (ii) their reasons for why they like or do not like the different solution features presented in the example solution artifacts in the concrete context. When analyzing the TAs’ goals, open coding [ 86 ] was used to generate initial categories that were constantly compared to the new data and refined by the entire research team to arrive at a final set of categories. After developing coding categories, coding was done by one researcher (SL), with about 1/3 of the codes checked by other researchers who independently assigned the codes. Any disagreements were discussed by the researchers until full agreement was established. Approximately 7% of the coding was modified in this process. To answer research question (1b), the numbers of TAs who identified the goal of developing an expert-like problem-solving approach along with the numbers of TAs who identified other goals in each context were counted. To answer research questions (2a) and (2b), we focus on feature clusters C1 to C3 in Table 2, which relate to the goal of developing an expert-like problem-solving approach. For each cluster, we identified whether each TA (i) notices/values these features in the solution artifacts provided (question 2a) and (ii) makes use of these features in their own solutions (question 2b). For question (2a), to portray TAs preferences of features before being influenced by their peers, we focused on their answers from the pre worksheets, yet, as mentioned previously, we used TAs answers from the post worksheets (where they were
why they would or would not include particular design features in the example solutions they present to students). We expected the goals that the TAs express might differ between the general and the concrete contexts. While in a general context instructors might express commitment towards possibly conflicting goals, the actual practice requires them to resolve such conflicts, exposing their priorities, some of which the instructor might not be conscious of (i.e. depth vs. time concerns, etc.). In addition, the design of the solution artifacts used in the concrete context aimed to elicit possibly deeply held ideas regarding “developing an expert-like problem-solving approach”. Such ideas are likely to be uncovered in the general, open ended context only if they are on top of the interviewees’ minds. We hypothesized that if such ideas are implicitly held, the artifact comparison technique may allow the TAs to articulate these ideas since the solution artifacts differed from each other in the extent to which they reflect a systematic decision making process characteristic of expert problem solvers. As discussed above, the goal categories were developed via an emergent coding procedure. The goals are summarized in Table 3 and then discussed in more detail in the following paragraphs. Note that some of the goals were only present in one of the two contexts (either general or concrete). The first two goals (G1 and G2) in Table 3 relate to expected performance of the students. Thus, we will refer to them as learning goals. The additional goals (G3 to G7) represent other considerations that the TAs expressed. These include considerations intended to support students’ learning process (G3 to G5) and pragmatic concerns regarding time and student grades (G6 and G7).
Learning goals G1 – physics understanding: An example solution should help students construct content specific physics understanding. This goal is expressed in both the general and concrete contexts, by 23 and 18 TAs, respectively. For example, the example solution should help to clarify concepts (e.g., “Examples make abstract concepts more concrete by showing how the concept is applied.” (TA4, general context)), relate concepts to problem solving (e.g., “It is most important to solve problems during the lecture, so that the students can relate the problems to the concepts they are being taught” (TA3, general context)), expand the breadth of possible approaches to solve a problem (e.g., “knowing how to do things several ways helps” (TA15, concrete context)), and refine understanding (e.g., “They [students] should be given solved problems after the homework is due but before the test so that they can prepare for the test and better understand where they may have gone wrong in their approach” (TA21, general context)). The TAs also believed that a good solution should help students focus on concepts rather than equations (e.g. "students should learn and memorize concept, not just equations so they can apply them" (TA2, concrete context)). All 18 TAs who mentioned this goal in the concrete context also mentioned it in the general context.
G2- PS approach: An example solution should help students develop an expert-like problem- solving approach. As we hypothesized above, the artifact comparison technique helped the TAs to recall and articulate this goal: it was expressed by few TAs in the general context (by 5 TAs) and by many in the concrete context (by 17 TAs). For example, in the general context, a TA expressed that “I think seeing problems in lecture provides students with the general idea of how to approach problems” (TA15, general context). When discussing the valued design features in an example solution (concrete context), the TAs believed there are several tools that should be included in the example solutions because these tools facilitate desired
thinking processes that are characteristic of expert-like problem solving. For example, a diagram “allows you to visualize [the problem]” (TA1, concrete context), and doing a unit check at the end “allows students to evaluate their final answer-does it make sense” (TA5, concrete context). In total, this goal is expressed by 18 unique TAs (in either the general context, the concrete context, or both.)
Other considerations G3 – emotional engagement: This goal is expressed by 9 unique TAs in total. It is expressed to the same extent in both the general context (by 5 TAs) and concrete context (by 6 TAs). However the meaning in these two contexts differs slightly. In the general context it was expresses in terms of students' confidence: Example solutions can be used to motivate students or to prevent student frustration. For example, “[After students seeing an example solution to a similar problem in the lecture,] when they try the [homework or exam] problem on their own, they can reference the different problem solving strategies. In this way, they aren't just starting blindly and they won't get as frustrated.” (TA5, general context); “It is also useful to see the abstract concepts used in a practical way in order to give the students motivation to learn the material” (TA6, general context). In the concrete context it was expressed in terms of maintaining students’ interest. For example, some TAs explained that they don’t like to have a detailed solution “[because students] won’t have patience to finish it” (TA16, concrete context). Other explained that the solution should “easily explain concepts without scary math” (TA6, concrete context).
G4- cognitive engagement: This goal is also expressed to a much lesser extent in the general context (by 6 TAs) as compare to the concrete context (by 21 TAs), and the meaning in these two contexts differs slightly. As described in the general context, example solutions should be presented at a proper time to engage students in cognitive processing. For example, “since everybody has to attend test and they tried very hard to solve that problem, so this [after the test] is a best time to explain some common mistakes” (TA10, general context); “Solutions can be useful only when the problems have been considered. If we give them the solution during lecture or before homework, they will copy it and it's totally useless” (TA22, general context). As described in the concrete context, example solutions should be communicated in a manner that allows students to follow it. Thus, solutions should be “easy to understand” (TA17, concrete context) and avoid the situation where “someone who is lost could not follow this” (TA6, concrete context). We believe that observing concrete artifacts that are less and more easy to follow raised TAs awareness to this aspect. In total, this goal is expressed by 22 unique TAs (in either the general context, the concrete context, or both.)
G5 – setting standards: This goal is expressed only in the concrete context. It is expressed by 12 TAs. There are some features that the TAs like because they are considered as the standard for an adequate solution. For example, the solution should include the solution process (e.g., "always tell the students to show work” (TA2)), be efficient (e.g. “physics is straight, it should be solved in the most simple way” (TA22)), and orderly presented.
G6 – saving time: This goal is expressed only in the concrete context. It is expressed by 5 TAs. Some TAs like a concise solution because a short solution "saves time" (e.g., TA3, TA 20). One TA explicitly points out that “[a solution with fewer steps] can save time in exam”,
C3- checking of solution. The results to questions (2a) and (2b) will be discussed three times: once for each of the three problem solving clusters.
1. Features Related to Initial Problem Analysis (C1)
(1) Question (2a – C1): Do TAs notice and value design features related to C1? Providing a schematic visualization of the problem (F1) and providing a list of knowns/unknowns (F2) are the features that relate to the explication of the initial problem analysis stage in a prescribed problem solving model. F1 is one of the most mentioned features (13 out of 24 TAs). F2 was mentioned by 9 TAs (the median for all features). As FIG. 9 shows, these features were valued by almost all TAs who mentioned them. Only one TA expressed that he didn’t like to provide a list of knowns/unknowns because it encourages students to solve problem via mindless plug and chug. Other TAs valued the list of knowns/unknowns because it “gives an idea of what you have and what you need.” Although all TAs who noticed F1 (visualization) valued it, different TAs had different ideas about the preferred visualization shown in FIG. 10. Nine out of thirteen TAs distinguished between the qualities of diagrams, with 6 of them preferring a detailed drawing as presented in solution 3. Most of the TAs did not articulate why the detailed diagram was better than the others. TAs who chose the less detailed diagrams in solution 1 and/or 2 explained, for example, that they didn’t like diagram 3 because “complicated diagrams can be confusing” (for example, the arrows in diagram 3 could be confusing to the students because they are used to represent both acceleration and velocity).
(2) Question (2b– C1) Do TAs use design features related to C1? Examination of TAs’ own solutions (which 23 TAs provided) indicates that features related to initial problem analysis are surprisingly more prominent in TAs actual practice than in their account of liked features. All TA solutions included a diagram and half included a list of knowns. A possible explanation resides in the fact that all three solution artifacts included a drawing, which made this a less noticeable feature within the artifact comparison technique.
2. Features Related to Solution Construction (C2)
(1) Question (2a – C2) Do TAs notice and value design features related to C2? Six features (F3, F4, F5, F6, F10, F12, see Table 2 in the Methodology Section) relate to the solution construction stage in a prescribed problem solving model. Based on Reif’s [ 13 ] representation of problem solving as a decision making process, the major choices a person makes in a solution process involve defining sub-problems: intermediate variables and principles to find them. These two aspects are evident in features F4 (Explicit sub-problems are identified), and F6 (Principles/concepts used are explicitly written). We grouped them under "Choices made" group. While F4 and F6 describe the major choices one makes, F (Providing a "separate" overview) and F5 (Reasoning is explained in explicit words) describe the solver’s reasoning underlying these choices. We grouped them under "Reasons for choices (additional explanations)" group. We note that this reasoning is guided by the solver’s general perception of the framework within which choices are made (e.g., as a
process that involves choosing between alternatives, or arriving at identified goal in a backward manner) represented in F10 (Providing alternative approach) and F12 (Forward vs. backward solution). FIG. 11 shows that features related to reasons for choices (F3 and F5) were the most noticed ones, but even those were noticed by less than half of the TAs (9 and 11 TAs, respectively). More than half of the TAs who noticed feature F3 thought that it was best represented in solution 3 (FIG. 4). Indeed solution 3 describes a complete overview of how the problem should be broken into sub-problems and explains the principles applicable in each of the sub-problems at the very beginning. As for feature F5, more than half of the TAs who noticed this feature thought that it was best represented in Solution 2, shown in FIG. 3. This solution identifies the goal of each sub-problem and provides justification for the principles separately as the progress of the solution. Although most TAs did not explicate why one presentation of F5 is better than the other in the worksheets, in the whole-class discussion several TAs raised their concerns that students may not have the patience to read the whole chunk of text at the beginning of solution 3. Students may simply ignore all the explanations in the first part and jump directly into the second part with equations. Reasoning that is presented beside the equations, as in solution 2, makes it easier to reference and students are more likely to process the information better. As shown in FIG. 11, we find that F3 and F5 were valued by most TAs who noticed them. The TAs believed that these two features play an important role in example solutions because they make the solution process clear and make the solution easier to follow. The TAs also believed that these features help students understand the internal thinking process that the instructor went through when solving the problem and facilitates better transfer to other problems. Except for minor concerns, such as “overdoing the motivations can lead to undesired chunks of text”, which was the major reason why a few of the TAs expressed a conflicted preference, these features were generally valued by TAs. Features 4 and 6, which explicate the choices made , were noticed by only a few TAs ( and 5 TAs, respectively), although they were valued by all TAs who noticed them. One TA explained that “I enjoy this feature [F4] because it helps set up a logical progression of the problem”; other TAs explained their preference towards F6 in that “the concepts may be more important than the answer” or “if we can use less math, I think we should do that, so students focus on physics”. Regarding the framework within which choices are made , 4 of the 5 TAs who noticed F (providing alternative approach) liked this feature, explaining, for example, that “this [feature] demonstrates how to develop an expert knowledge structure and how it makes the problem much simpler.” One TA was conflicted about this feature, as presenting an alternative approach “could possibly confuse students.” As for F12 (backward vs. forward solution), only one TA noticed it as an important consideration in the design of a solution.
(2) Question (2b– C2) Do TAs use design features related to C2? Examination of TAs’ own solutions indicates a discrepancy between their self-reported preferences and their actual practice. Although F3 and F5 were valued by 7 and 10 TAs, respectively, only 3 out of 23 TAs provided some outline of the sub-problems (F3) either at the very beginning or along the solution progression, and only 6 of the 23 TAs provided any justification for the principle(s) used (F5). None of the TAs presented a solution in which the goals for each sub-problem were clearly stated (F4). None of them provided an alternative
C. Relationship between design features and goals
In the previous section, we examined the extent to which TAs value/use features that the literature suggests as supportive to the goal of helping students develop expert-like problem- solving approaches. Here, we look at the extent to which the TAs themselves connect these features to that goal. Moreover, we investigate how other goals and considerations interfere with the use of features that help students to develop an expert-like problem-solving approach. In order to do so, we first look at question (3a), examining all the features and their relationship to the goals that the TAs expressed.
1. Question 3a: What design features of example solutions do TAs perceive as supporting different goals/considerations?
To get a somewhat more global picture, the features' (grouped into clusters as described in Table 2) relation to the goals is portrayed in figure 13. The x-axis states the various goals/considerations that the TAs mentioned. For each of these goals, the number of TAs who perceive different features as supportive or contradictive to the different goals are plotted. The height of each bar indicates the number of TAs who noticed at least one feature in that particular cluster and believed that the feature(s) support (represented as positive) or contradict (represented as negative) that goal. For example, if a TA noticed F3 and F10 – both from the C2 cluster – and valued them because s/he believed these features can help students develop expert-like problem solving approach (G2), this TA is represented as one single unit in the gray positive chunk of the G2 bar. On the other hand, if another TA noticed F10 and dislike this feature because s/he believed that this feature would make the solution confusing, therefore hindering goal G4, this TA is represented as one single unit in the gray negative chunk of the G4 bar. It is possible that a single TA may consider features from different clusters as supportive to the same goal/consideration. Therefore, for each goal, the maximum value on the vertical axis could potentially sum up to 120 (24 TAs times 5 clusters). As one might expect, the analysis (C1), construction (C2) and checking (C3) clusters - clusters that are aligned with the prescribed problem solving model by Reif - are indeed those that TAs recognize as supporting the goal of helping students develop an expert-like problem-solving approach (the G2 shown in FIG. 13). Moreover, these three clusters are also valued by the TAs because they support some other goals/considerations. For example, the solution construction (C2) cluster is the most prominent cluster in helping students construct content specific physics understanding (G1- physics understanding) and is also prominent in promoting “ cognitive engagement ” (G4). In addition to clusters C1, C2, and C3, another prominent feature shown in FIG. 13 is the extended details cluster (C4), which is considered as disadvantageous in regard to many considerations (such as G3 – emotional engagement , G6 - saving time and G7 – preventing mistake ), and in some cases both positive and negative (such as G5- setting standards and G4- cognitive engagement ). For example, although a detailed solution may make it easier for students to follow, it could also work in the opposite way and make the students lose the thread more easily.
We now turn to examine whether the above portrayal of goals and practices challenges the materialization of the goal of helping students develop an expert-like problem-solving approach.
2. Question 3b: Where do conflicts exist (if any)?
The above portrayal of relationships between the features and goals, in particular, the positive occurrence of cluster C2 ( solution construction) and the frequent negative occurrence of cluster C4 ( extended details ), suggests a possible conflict between design features supporting the development of an expert-like problem-solving approach (G2) and design features supporting other goals/considerations. While C2 is perceived as one of the prominent clusters supporting goal G2, it usually requires a longer length and more details, which is represented by the cluster of extended details (C4). However, C4 is considered as disadvantageous in regard to many considerations. Such finding suggests a possible challenge to materialize the goal of developing an expert-like problem-solving approach and other goals in a coherent manner. In order to examine whether such a conflict actually exists within individual TAs and to get more insight into the nature of the conflict, TAs’ preferences for cluster C2 are compared to their preferences for cluster C4. In particular, we focus on features F3 ( providing a "separate" overview ) and F5 ( reasoning is explained in explicit words ) in the solution construction - C2 cluster and features F7 ( thorough derivation ) and F8 ( long physical length ) in the C4 cluster. F3 and F5 were chosen because they are the features in cluster C2 that most require extended details. F9 (details without which the solution is still technically correct) was excluded from the features in C4 (leaving F7 and F8) because F9 is already stated in a negative manner. In particular, to gain insight into how TAs resolve the conflict we: (i) identify how TAs holding conflicting values between different features (even though they may not be aware of it) resolve this conflict in their actual practice, (ii) identify the direction in which TAs shift their preferences for a single feature between the pre and the post worksheets, (iii) identify TAs resolution for conflicting preferences and concerns regarding a single feature that they were aware of and raised in either one of the worksheets itself. In total there were 13 unique TAs who fell into one or more of these categories (with 9, 3, and 6 TAs falling into categories (i), (ii), and (iii), respectively.) Each category is discussed in detail in the followings. For (i), out of the 14 TAs who mentioned at least one feature from each of these two groups (F3/F5 vs. F7/F8), 9 of them were found to hold a conflicting value: while they valued F3 ( reasoning ) and/or F5 ( overview ) in the C2 cluster, they dislike the feature(s) F ( derivation ) or F8 ( long ) in the C4 cluster. Examining their own solutions indicates that except for one TA’s solution which is detailed, most of these TAs’ solutions were concise, and there is little, if any, F3 ( reasoning ) or F5 ( overview ) found in them. Thus, when TAs were not aware of the conflict, in most cases their resolution gave up features supportive of expert like problem solving. However, when TAs were aware of the conflict, the resolution may have taken a different direction. For (ii), it is observed that after the group discussion, three TAs indeed re- considered their former preferences regarding the design of problem solutions and explicitly changed their preferences from concise solution to thorough presentation after the group
