1
Evaluation for C&I 335’s lab section at University of Illinois Champaign
Mei-Li Shih
Department of Educational Psychology
EPSY 474: Evaluating Learning Technologies
Dr. Bullock
University of Illinois at Urbana-Champaign
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Program Background and Evaluation Rationale - Study Guide | EPSY 474, Papers of School management&administration
Material Type: Paper; Professor: Bullock; Class: Evaluating Learning Technology; Subject: Educational Psychology; University: University of Illinois - Urbana-Champaign; Term: Unknown 2007;
Typology: Papers
Pre 2010
1 / 29
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Program Background and Evaluation Rationale - Study Guide | EPSY 474 and more Papers School management&administration in PDF only on Docsity!
Evaluation for C&I 335’s lab section at University of Illinois Champaign Mei-Li Shih Department of Educational Psychology EPSY 474: Evaluating Learning Technologies Dr. Bullock University of Illinois at Urbana-Champaign
Content Table
- Section I: Abstract.................................................................................................. ……….
- Section II: Evaluation Questions
- Section III: Program Background and Evaluation Rationale..............................................
- Stakeholders………………………………………………………………………….
- Evaluator……………………………………………………………………………..
- Evaluation Rationale…………………..…………………………………………..
- Evaluation Standards..……………………………………………………………….
- Section IV: Evaluation Methods and Procedure Section..................................................
- Introduction.......................................................................................................... …
- Evaluation Methods
- Pilot Study, Focus Group and Survey Administration.............................................
- Sampling……………………………………………………………………………
- Section V: Evaluation Plan and Procedures......................................................................
- Gantt Chart………………………………………………………………………….
- Data Analysis………………………………………………………………………
- Section VI: Evaluation Results
- Pilot Study................................................................................................................
- Focus Group.............................................................................................................
- Survey
- Section VII: Recommendations
- References.........................................................................................................................
- Appendix A: Cost-Benefit and Cost-effectiveness
- Appendix B: Consent Letter
- Appendix C: Survey Sample.............................................................................................
Section II: Evaluation Questions y What were students’ technology skills and knowledge before they enrolled in the class? What hardware and software were students already familiar with? y What new technology skills have students learned in the class during the lab section? y What were the most and least useful technology software applications that students learned in the computer lab? And why? y What technology skills would students adapt into their multimedia final project? y What were students’ instructional needs from the teaching assistants in the lab?
Section III: Program Background and Evaluation Rationale The site for my evaluation project was the 335 class in the department of Curriculum and Instruction at the University of Illinois at Urbana-Champaign. The course C&I 335, Content Area Applications of Educational Technology , was a required course for college students who were getting a Teacher Education Certification for elementary education. More information about the TEC certification program can be found at http://courses.uiuc. edu/cis/programs/urbana/2004/fall/undergrad/education/elem_school_tch.html. Most of the students in the class were juniors and seniors, and a few of them were master students. These students were usually called “pre-service teachers” or “teacher candidates”. The class C&I 335 was designed to expose students to curriculum content, application and technology integration. Technology continues to pervade all aspects of the 21 st^ century. Technology integration of content material is a necessary skill that must be developed for educators of every grade level from kindergarten to higher education. The purpose of this class was to develop an understanding of technology tools and their application. Students were supposed to learn the educational purpose of many technology skills in their computer lab for the course. Some of the technologies students learned how to use were digital cameras, digital videos, animation software, and Power Point, as well as how to build their own electric teaching portfolio and so on. In this class, students were required to attend one hour of face-to-face lecture and one hour of hands on lab sections every week. There were 75 students in the class during the fall 2004 semester and these students were divided into four lab sections. Each lab section had one teaching assistants who provided major assistance in the computer lab. According to the syllabus, the learning outcomes for this class were: y Examine issues related to the internet and the law. y Learn terminology, questions, content, and issues dealing with technology y Formulate a personal definition and understanding of teaching with technology
had learned in the hands-on lab section. According to the findings of this evaluation, they have learned what technology application may be useful to integrate in the elementary curriculum. In summary, this evaluation fitted all stakeholders’ needs. As a stakeholder myself, this was a significant evaluation for my previous and future teaching.
Evaluator: I was not a current teaching assistant in the class; therefore, this project was going to be an external evaluation. Also, I believe this was a formative evaluation because the primary purpose for this evaluation was to provide information for program improvement. The course and content already existed so that my project just evaluated students’ learning in the courses. As an experienced teaching assistant for the C&I 335 lab section, I was very familiar with the content of the course and technology software applications in the class. Although there was some bias on my part in that I hoped for favorable results, I didn’t see that as a negative. The results helped stakeholders to improve the quality of future course. The results had no impact on my position or the positions of those involved.
Evaluation Rationale : The reason I chose this program was because I was a teaching assistant in C&I 335 last year, and I am going to be a teaching assistant in the same class next semester. As a teaching assistant, I thought it was important to understand how the class runs with respect to students’ technology skills entering the course, technologies learned throughout the course, and what kind of technologies students prefer using in their final projects (a possible indication of what kinds of technologies students feel were the most useful). This was important because I (and other TAs) needed to find out how to help students learn
effectively and efficiently and provided useful materials for them to refer back to when the situation arises later in their careers. I have taught C&I 335’s lab section for three semesters. Because of my teaching experience in the lab, I had many questions about how to improve my teaching and how to help students learn better. Since the C&I 335 lab section required many hands-on activities of a variety of different technologies, I wanted to know how much students have learned from completing the hands on activities in the lab. I know I am going to be a teaching assistant again for the same class in spring 2005. This evaluation project came at the perfect time. This evaluation project gave me a great opportunity to discover answers to my questions so that I could make recommendations for changes for the upcoming semester. In this way I could improve my own teaching style, and I could help other TAs know what to expect as they begin preparing to teach the lab section of C&I 335. The findings not only answered my questions I’ve had in previous semesters, but also helped me to prepare to be a better TA for next semester.
Evaluation Standards : This project met the four standards of evaluation. Utility standards: The Utility standards were intended to ensure that an evaluation would serve the information needs of intended users. For this evaluation, I informed the professor who was teaching the class. She was in agreement with my evaluation project and was happy to see the results. The professor was one of the stakeholders in this project. She was excited that the findings would be useful for her future resources as a professor in the class. Timing was another concern for the utility standard. The evaluation should be done when the stakeholders need the results. According to the EPSY 474 course requirement, I had to finish this evaluation by the end of this semester (Fall 2004). I didn’t believe that
Section IV: Evaluation Methods and Procedure Section Introduction : I planned to use mixed methods, which includes quantitative and qualitative methodologies for this evaluation. This mix-method research was based on Jennifer Greene’s theory (Fitzpatrick 2004). This evaluation project included both a survey and focus group. Indeed, the overall evaluation focused on three approaches: management-oriented, consumer-oriented and participant-oriented evaluation approaches. First, it was management-oriented in that this evaluation tried to meet stakeholders’ needs. Second, it was also a consumer-oriented approach in that this evaluation tried to develop a class and help students (primary consumers) to learn with hands-on curriculum. Finally, it was a participant-oriented evaluation approach in that this evaluation tried to respond to the natural ways in which people assimilate information and arrive at understanding. To sum up, all three approaches, management-oriented, consumer-oriented, and responsive- oriented evaluation approach (Stake’s theory) have strengthened this evaluation.
Evaluation Methods : My evaluation methods were both a focus group and survey. There were two purposes of a focus group. First, the focus group helped me to make sure the survey questions were understandable, and then I was able to refine the survey questions and instruments. Second, a focus group helped me to get more detailed information that a survey could not do for the evaluation questions. There were five questions on the survey. The first question was a multiple-choice question. Participants circled the answers to what hardware and software they were already familiar with before they entered the course. The questions two to five were open-ended questions. The five evaluation questions were answered accordingly on the
survey. Each evaluation question was answered from the survey by numeric order. Pilot Study, Focus Group and Survey Administration : I received notice of IRB approval on 10/08/04 from the University of Illinois at Urbana Champaign. All of the evaluation project, consent letter and survey questions were reviewed and approved. (Please see consent letter at appendix B and survey sample at appendix C). Pilot Study: A pilot was the important first step to make sure my project and survey questions were clear to students. I also got a chance to know students questions during the pilot study. This helped me to overview my evaluation questions and methods. Focus Group: I conducted a focus group before I distributed surveys to the participants. According to the focus group’s responses, I revised the survey as necessary. I was looking for 6- participants out of 75 students for a focus group. The focus group met about one hour on the UIUC campus. To make sure students felt comfortable to express their opinion, I did not plan to audio tape the conversation during the discussion. I only took notes during the discussion. I provided drinks during the discussion. I also provided $10 as incentives for any participant who volunteered to attend the focus group. Survey: Since I was able to visit the evaluation site, I conducted a site survey. I visited the class during the computer lab section. I visited the site at the beginning of the lab section. The students had enough time to complete the survey at the beginning of the class time. I distributed the consent letter to students and explained the purpose of this evaluation before they started the survey. I made sure that students understood their participation was voluntary. The students were also told that all of the information that was obtained during
Section V: Evaluation Plan and Procedures This evaluation has taken several steps to complete through the Fall 2004. The following Gantt chart shows how I developed a management plan by each step.
Action^ 9/6 to 9-129/13 to 9-199/20 to 9-269/27 to 10/310/4 to 10/1010/11 to 10/1710/18 to 10/2410/25 to 10/3111/1 to 11/711/8 to 11/1411/15 to 11/2111/22 to 11/2811/29 to 12/ Selected project topic and began project development Completed evaluation questions Submitted and revised the background section Submitted IRB and approved. Designed survey questions. Submitted and revised evaluation design section Met participants and conducted the pilot study and focus group. Distributed Survey Submitted and revised evaluation plan /procedures section. Complete evaluation results, recommendations. Final project presentation.
This evaluation was my individual project. The timetable above not only helped to keep me on track but also fitted the requirements of the assignment in this course. I did not have any major problems while conducting this evaluation. Before I conducted the focus group and distributed the survey, I had a chance to go to
my evaluation site and met all participants during class time on Oct 26, 2004. I explained to all students who I am and the purposes of the evaluation. I described the benefits of this evaluation that will help this course, TAs and themselves. Students were able to ask any questions to me face to face. If they were interested in joining the focus group, they signed a paper and gave it to me at the end of class. I was looking for 8 out of 75 students to participate in the focus group. Each student would receive $10 for participating in the focus group. Unfortunately, there were only four students who signed up for the focus group. I was upset that there were only four students willing to participate in the focus group; however, the result of the focus group was great and very useful. I emailed students who signed up to participate in the focus group and scheduled a meeting time with them. I conducted a pilot study with the first student. I met her individually in the Illini Union at University of Illinois at Urbana Champaign. The pilot study took approximately one hour (12:00-1:00PM) on Oct 28, 2004. The pilot study was helpful. I was able to sit down and talk about my study with a student for the first time. The pilot study also helped me to know what questions I could ask students during the focus group meeting. At the end of meeting, the student received $10 for participating in the pilot study. The focus group was after the pilot study. The focus group was conducted in a coffee shop at Krannert Art Museum at the University of Illinois-Urbana Champaign. The focus group met approximately one hour (2:00-3:00PM) on Oct 28, 2004. I bought each student a drink. Each student also received $10 as incentive for participating in the focus group at the end of meeting. I wrote important notes during the discussion and re-organized the information at home. According to the focus group’s feedback, I revised the survey questions that were distributed during the four lab classes on the week of Nov 1- Nov 7,
The survey was distributed in the students’ computer lab classroom, Room 16, in the
Section VI: Evaluation Results Pilot Study : For the pilot study, I met one student and talked about an overview of my project. The student tried the survey questions during the meeting. I walked through each question with her and made sure she had no problem understanding the survey questions. I also tried to ask additional detail questions on each survey question. For example, “Do you even go to the TA’s office when you need help?”, “How many hours do you usually spend for this class every week?”. The student felt the TA’s office hours were not useful to her because she had a class at that time. She usually emailed the TA whenever she had questions. She mentioned that the TA was very reachable through the email. She said she usually spent 2-3 hours on the assignment each week. She thought the assignments were not difficult but there were too many assignments in this class. This pilot helped me to understand the student’s problems in the lab. Her responses and feedback also helped me to organize several additional questions for the focus group. Focus Group : There were three students who joined the focus group. Just as I had done in the pilot study, I asked students to do the survey questions and walked thought each question with them. I made sure students understood the survey questions and provided the correct answers for this study. Students also talked through their answers during the conversation. According to my experience from the pilot study, I paid attention to students’ problems and needs from the TA in the computer lab. Overall, they liked the lab section. The TA was very helpful but they needed more time on the lab. Some of computer software such as Photoshop was not available in their computer at home. Because they were not able to practice at home, they really needed extra time to learn during the lab time. They thought a handout would be very helpful for them to learn unfamiliar computer software outside of class. In addition, TAs were very helpful but the TAs were not able to help every
individual during the one hour lab hour. Students mentioned it would be nice if they could have one or two more TAs in the class. Another big concern was the explanation for using particular computer software in the education file. For example, they were not sure how to integrate Photoshop into elementary curriculum. They wished there were an example or guideline available for them when they were learning new computer software. The focus group spent approximately one hour. I had a great conversation with the students. I combined both the focus group information and the following survey results for the evaluation findings. Survey : There were five questions in the survey. The following charts and graphs show the survey result for each question. Question 1 : What was the software application most students already knew before they enrolled in the class? The major two computer software programs that students were already familiar with were Word (70/70, 100%) and PowerPoint (58/70, 82.9%) Knew Word
Frequency Percent Valid Percent
Cumulative Percent Valid yes 70 100.0 100.0 100. Table 1: The number and percentage of students who were familiar with Word before enrolling in the class knew PowerPoint
1258 17.182.9 17.182.9 (^) 100.017. 70 100.0 100.
no yes Total
Valid Frequency^ Percent^ Valid Percent
CumulativePercent
Table 2: The number and percentage of students who were familiar with PowerPoint before enrolling in the class
yes^ no
Have learned front page noyes
Have learned photoshop
Figure 3: Students have learnedFront Page Figure 4: Students have learnedPhotoshop
Question 3 : What were the most and least useful technology software applications that students are learning in the computer lab? And why? The two major most useful computer software were Excel (27/70, 38.6%) and Front Page (27/70, 38.6%). The reason they were useful was because teachers could use Excel as a grade book and use Front Page to create a class website for the class. The class website would help teachers to communicate with students and parents outside of class. useful/less excel 3332 47.145.7 47.145.7 47.192. 5 7.1 7.1 100. 70 100.0 100.
not mentionuseful least usefulTotal
Valid Frequency^ Percent^ Valid Percent
CumulativePercent
Table 5: The number and percentage of most useful software as Excel. useful/less front page
3527 50.038.6 50.038.6 50.088. 708 100.011.4^ 100.011.4^ 100.
not mention usefulleast useful Total
Valid Frequency^ Percent^ Valid Percent
CumulativePercent
Table 6: The number and percentage of most useful software as Front Page.
notmention usefulleast useful
useful/less excel notmention usefulleast useful
useful/less front page
Figure 5: Most useful software I Figure 6: Most useful software II The least useful computer software was: Photoshop (37/70, 52.9%). Students thought Photoshop was too complicated and students had no idea how to integrate Photoshop into elementary curriculum.
useful/less photoshop
26 37.1 37.1 37. 7 10.0 10.0 47. 37 52.9 52.9 100. 70 100.0 100.
not mention useful least useful Total
Valid Frequency^ Percent^ Valid Percent
CumulativePercent
Table 7: The number and percentage of least useful software as Photoshop.
notmention usefulleast useful
useful/less photoshop
Figure 7: Photoshop as the least useful software