Using Rubrics for Program-Level Assessment of Student Learning Outcomes, Study Guides, Projects, Research of Physical education

Download Using Rubrics for Program-Level Assessment of Student Learning Outcomes and more Study Guides, Projects, Research Physical education in PDF only on Docsity!

U SING R UBRICS

TO A SSESS S TUDENT L EARNING O UTCOMES

AT THE P ROGRAM L EVEL

OFFICE of INSTITUTIONAL RESEARCH and ASSESSMENT

July 2017

Table of Contents

• Using Rubrics to Assess Student Learning Outcomes at the Program Level
• What are Rubrics?
• What Do Rubrics Consist of?
• What are the Benefits and Challenges of Using Rubrics?
• How is a Rubric Created and Used in Assessing Program-Level Student Learning Outcomes?
• More Questions and Answers about Using Rubrics in Program-Level Assessment of Student Learning Outcomes.......
• Examples of Rubrics..........................................................................................................................................................
• Contact Information
• Appendix A - Sample Rubrics | Papers
  • Rubric for a Research Paper in the Humanities.................................................................................................
  • Rubric for a Paper in the Social and Behavioral Sciences..................................................................................
• Appendix B - Sample Rubrics | Presentations
  • Scoring Rubric for General Oral Presentation
  • Poster Presentation Peer Scoring Rubric
• Appendix C - Sample Rubrics | Arts and Humanities
  • Rubric for Visual Arts
  • Rubric for Digital Media Project
• Appendix D - Sample Rubrics | Science and Math
  • Rubric for Conducting an Experiment in the Lab
  • Evaluation Rubric for Undergraduate Research Project in the Sciences
  • Science Lab Report Evaluation Rubric
  • Evaluation Rubric for Mathematical Proofs
• Appendix E - Sample Rubrics | Graduate Work
  • Evaluation Rubric for Thesis in Linguistics.........................................................................................................
  • Rubric for a Dissertation....................................................................................................................................
  • Thesis Proposal Rubric.......................................................................................................................................
• Appendix F - Sample Rubrics | Miscellaneous
  • Undergraduate Internship Rubric
  • A Rubric for Rubrics

• Checklists are dichotomous (e.g. Yes/No; Met/Not Met) and easy to use; however, they may not provide substantive information regarding the performance levels.
• Rating scales identify a range of performance without specific descriptors for each performance level (e.g. Exceeds Expectation/Meets Expectation/Below Expectation). They may not provide sufficient information for raters to score consistently and the feedback to students may not be specific enough for revision or future growth.

Rating Scale Example: Computer Program Quality Assessment Expected Learning Outcome: The student will write efficient, documented, error-free computer programs that meet the specifications. Criteria for Success: A maximum of one item is rated as “Below Expectations”.

Computer Program Below Expectations=

Meets Expectations=

Exceeds Expectations=

Comments

Achieves what it was designed to do

Operates without errors Source code is efficient Source code is well- documented Exceeds Expectations = Performance is above the expectations stated in the outcomes. Meets Expectations = Performance meets the expectations stated in the outcomes. Below Expectations = Performance does not meet the expectations stated in the outcomes.

Interactive Presentation Checklist Yes No Time Management

Organization

Resource use

Audience Awareness

Aesthetics

11. Did I hand out and use materials that were ascetically pleasing?
12. Did I present my material in a way that suited my audience?
13. Did I present myself in a professional way in my dress and grooming?
14. Did I make frequent eye contact with my audience?
15. Did I vary my voice to suit my presentation?
16. Did I use different media to present my information?
17. Did I use the most important media for the kind of information I presented?
18. Was the flow of my presentation and material logical and smooth?
19. Were all the presented materials well-organized and readily available?
20. Did I use my time at an even pace, completing all sections of the presentation?
21. Did I set up and begin promptly?

• Analytic scales articulate specific levels of performance for each individual criterion. Scoring using analytic scales is typically more consistent and specific areas of growth can be identified. The analytic scales take more time to develop.

What are the Benefits and Challenges of Using Rubrics?

Benefits –

• Establishes shared expectations and assessment practices, especially when faculty members collaborate to develop them
• Can be used to evaluate student work consistently
• Make it more efficient to assess multifaceted examples of student work or performance

Challenges –

• Take time and effort to develop
• Need to include descriptions of specific criteria, and preferably exemplars of student work to benchmark different levels of performance
• May be a less precise measure because of the broad range of performance within each specified level

Analytic Scale Rubric for Evaluating a Portfolio 1 2 3 4 Score

Portfolio Requirement

Poor ePortfolio is missing more than 4 minimum requirements as stated in the syllabus.

Fair ePortfolio is missing 3 minimum requirements as stated in the syllabus.

Good ePortfolio meets all minimum requirements as stated in the syllabus.

Exceptional ePortfolio meets all minimum and above requirements as stated in the syllabus.

Creative use of Technology

Poor No use of graphics, Internet resources, photographs, sound and /or video to enhance ePortfolio and reflective statements.

Fair Little use of graphics, Internet resources, photographs, sound and /or video to enhance ePortfolio and reflective statements.

Good Some use of graphics, Internet resources, photographs, sound and /or video to enhance ePortfolio and reflective statements.

Exceptional Good use of graphics, Internet resources, photographs, sound and /or video to enhance ePortfolio and reflective statements.

Artifacts

Poor Most artifacts and work samples are unrelated to the purpose of the course and portfolio.

Fair Few artifacts and work samples are related to the purpose of the course.

Good Most artifacts and work samples are related to the purpose of the course and portfolio.

Exceptional All artifacts and work samples are clearly and directly related to the purpose of the course and portfolio.

Organization & Writing

Poor The text has many errors in grammar capitalization, punctuation, and spelling requiring major editing and revision.

Fair The text has errors in grammar, capitalization, punctuation, and spelling requiring editing and revision.

Good The text has a few errors in grammar, capitalization, punctuation, and spelling requiring editing and revision. Easy to read and navigate.

Exceptional The text has no errors in grammar, capitalization, punctuation, and spelling. Easy to read and navigate.

Reflections

Poor Considerable difficulty in expressing reflections demonstrates level of students' progress and knowledge development. Reflections incorporate the what, so what and now what in artifacts.

Fair Difficulty expressing and reflecting demonstrates level students' progress and knowledge development. Reflections incorporate the what, so what and now what in artifacts.

Good Most reflections demonstrate students' progress and knowledge development. Reflections incorporate the what, so what and now what in artifacts.

Exceptional All reflections demonstrate students' progress and knowledge development. Reflections incorporate the what, so what and now what in artifacts

8. Aggregate the rating scores across the entire sample. Compare the results to the program’s target for performance on that learning outcome.

After implementing the rubric, continue to review the findings. In addition to proving a consistent method for assessing student work, rubrics can identify opportunities for program improvement. Trends uncovered through aggregated rubric scores can determine areas where students need additional instruction or support, as well as inform changes in the curriculum or how content is taught.

More Questions and Answers about Using Rubrics in Program-Level Assessment of

Student Learning Outcomes

Question: Do rubrics have to be created and used in assessing all student work to measure achievement of expected learning outcomes?

Answer : No. There are a number of different methods of assessing program-level student learning outcomes. However, there are advantages to using rubrics, including greater reliability and consistency in rating performance across the program, the ease with which scores can be aggregated across the program to do more sophisticated analysis, and the ability to examine different dimensions of performance as opposed to just an overall grade or score.

Question: Why can’t we just use average course grades to assess student achievement of learning outcomes?

Answer: Accreditors do not recommend using course grades to assess learning outcomes. They argue that course grades, especially at the undergraduate level, often include class participation, attendance, and other behavior not directly related to the learning outcomes. Further, average final grades provide no insights into relative strengths and weaknesses of students across the various components of the outcome. Carnegie Mellon University’s website offers a useful description of the differences between course grades and assessment of program-level learning outcomes. Instead, assess a specific assignment within one or more courses that maps to the learning outcome of interest.

Question: Is there ever a situation in which we can use average course grades as evidence of achieving program- level learning outcomes?

Answer: This may be possible at the graduate level if you can document that that 100% of the course grade was based on the quality of a paper, research project, or final exam that directly measured the learning outcome. But if a graduate program has a comprehensive or qualifying exam or dissertation proposal that measures the knowledge covered in the required core courses, why not just use the performances on those milestones as evidence of achievement of the outcome? Reporting on course grades would be redundant.

Question: We mapped our undergraduate level student learning outcome for research skills to the final paper in our capstone course, which reports the procedures and results of their semester-long project. The course instructor graded the papers measuring competency in each of the stages of the research process. Do we need to have additional faculty independently re-evaluate the papers using a formal rubric?

Answer: No. When there is such a close fit between the learning outcome and the work being used to assess it, the scores or ratings originally assigned by the instructor can probably be used as evidence as to whether students are achieving the outcome. Most of the time, no one is more qualified to evaluate the work than the person who taught the course. For the department’s annual Student Learning Outcomes Assessment Report, the instructor could prepare a brief report for the DUS or undergraduate studies committee that includes a description of the assignment, a copy of the rubric or grading scheme used, the results of the analysis of student performance, observations about strengths and weaknesses, and recommendations for improving student learning related to the research skills outcome.

Question: To assess our learning outcome related to critical thinking skills, we plan to sample final papers from a set of senior-level courses in our major and have them rated by two faculty members with a rubric designed to measure critical thinking. Some faculty have criticized this plan, saying that it is a poor use of time to re-grade student work and that averaging final course grades across the set of courses should provide sufficient data for this assessment exercise. In addition, concerns were expressed that having other faculty not associated with the course re-grade another faculty member’s papers sends a message of distrust and raises questions about their academic freedom to make judgments about student performance in their own courses. How do I explain this?

Answer: The papers will be used for a secondary analysis to assess whether students are achieving a single program-level outcome. This is not the same as having the papers completely re-graded by independent raters using the same criteria as the original instructor. The instructors no doubt evaluated the papers with additional expectations unique to their particular courses, such as specific subject matter knowledge. They might have taken critical thinking skills into account when grading the papers, but unless common criteria for this competency were used across instructors and assignments, aggregating the course grades will not provide very reliable information about whether seniors have achieved this specific skill.

In addition, it is important for all involved to understand that the assessment of learning outcomes is student-focused; it is not used to evaluate instructors. And, there is no reason to exclude the original course instructors from rating the papers for evidence of critical thinking skills, although it is a good idea have them rate papers that they have not already read for another purpose.

In selecting the courses and papers for use in this exercise, the raters will need to ensure that the different assignments provided comparable opportunities for students to demonstrate critical thinking skills. Raters should meet to reach a common understanding of the rubric and how it is to be applied, and to test the consistency of their ratings.

Examples of Rubrics

Examples of rubrics used by other institutions to evaluate different kinds of student work are provided in the Appendix. They can be used as models, or you can copy and adapt them for your own purposes. However, please be sure to acknowledge the source (listed under each matrix) of each instrument if you distribute it.

Another great source for rubrics is the American Association of Colleges and Universities’ (AAC&U) Valid Assessment of Learning in Undergraduate Education (VALUE) project. This set of 16 rubrics includes the following that are relevant for measuring a number of learning outcomes across majors:

Critical Thinking Oral Communication

Creative Thinking Quantitative Literacy

Inquiry and Analysis Problem Solving

Written Communication Integrative and Applied Learning

The VALUE rubrics, which are being used by hundreds of institutions, are free and can be downloaded from AAC&U’s website: http://www.aacu.org/value/rubrics.

Appendix A

Sample Rubrics | Papers

Rubric for a Research Paper in the Humanities

Excellent Good Acceptable Unacceptable Introduction Strong introduction of topic’s key question(s), terms. Clearly delineates subtopics to be reviewed. Specific thesis statement.

Conveys topic and key question(s). Clearly delineates subtopics to be reviewed. General thesis statement.

Conveys topic, but not key question(s). Describes subtopics to be reviewed. General theses statement.

Does not adequately convey topic. Does not describe subtopics to be reviewed. Lacks adequate theses statement. Focus & Sequencing

All material clearly related to subtopic, main topic. Strong organization and integration of material within subtopics. Strong transitions linking subtopics, and main topic.

All material clearly related to subtopic, main topic and logically organized within subtopics. Clear, varied transitions linking subtopics, and main topic.

Most material clearly related to subtopic, main topic. Material may not be organized within subtopics. Attempts to provide variety of transitions

Little evidence material is logically organized into topic, subtopics or related to topic. Many transitions are unclear or nonexistent.

Support Strong peer- reviewed research based support for thesis.

Sources well selected to support thesis with some research in support of thesis.

Sources generally acceptable but not peer- reviewed research (evidence) based.

Few sources supporting thesis. Sources insignificant or unsubstantiated.

Conclusion Strong review of key conclusions. Strong integration with thesis statement. Insightful discussion of impact of the researched material on topic.

Strong review of key conclusions. Strong integration with thesis statement. Discusses impact of researched material on topic.

Review of key conclusions. Some integration with thesis statement. Discusses impact of researched material on topic.

Does not summarize evidence with respect to thesis statement. Does not discuss the impact of researched material on topic.

Grammar & Mechanics

The paper is free of grammatical errors and spelling & punctuation.

Grammatical errors or spelling & punctuation are rare and do not detract from the paper.

Very few grammatical, spelling or punctuation errors interfere with reading the paper.

Grammatical errors or spelling & punctuation substantially detract from the paper. Communication Scholarly style. Writing is flowing and easy to follow.

Scholarly style. Writing has minimal awkward of unclear passages.

Word choice occasionally informal in tone. Writing has a few awkward or unclear passages.

Word choice is informal in tone. Writing is choppy, with many awkward or unclear passages. Citations & References

All references and citations are correctly written and present No errors in MLA style.

Rare errors in MLA style that do not detract from the paper. Scholarly style. Writing has minimal awkward of unclear passages.

Errors in MLA style are noticeable. Word choice occasionally informal in tone. Writing has a few awkward or unclear passages.

Reference and citation errors detract significantly from paper.

(Adapted from the University of Kentucky)

Rubric for Grading an Essay Exam

Criteria & Points Assigned

Missing or Serious Problems

Below Expectations

Meets Expectations Excellent Work (^) Points Earned

0 1 2 3

Relevance of answer to the question

The essay did not answer the question.

Answer is incomplete. Excessive discussion of unrelated issues and/or significant errors in content.

Answer is brief with insufficient detail. Unrelated issues were introduced and/or minor errors in content.

Answer is complete; sufficient detail provided to support assertions; answer focuses only on issues related to the question; factually correct.

Thoroughness of answer

None of the relevant details were included.

Serious gaps in the basic details needed.

Most of the basic details are included but some are missing.

Deals fully with the entire question.

Organization and logic of answer

Weak organization; sentences rambling; ideas are repeated.

Minor problems of organization or logic; Needs work on creating transitions between ideas.

Clear and logical presentation; good development of an argument; Transitions are made clearly and smoothly.

Mechanics of writing (spelling, punctuation, grammar, clarity of prose)

Major problems with mechanics of language; Awkward sentence construction; Poor or absent transitions; Frequently difficult to understand.

Frequent problems with mechanics of language; Occasional awkward sentences and poor transitions; reduce readability.

Clear, readable, prose. Good use of transitions; no problems with spelling, punctuation, or grammar.

(Adapted from the University of West Florida)

Appendix B

Sample Rubrics | Presentations

Poster Presentation Peer Scoring Rubric

Presenter’s Name: __

Poster #:

Poster Research Category:

Score Key: 0 = No Attempt 1 = Developing 2 = Competent 3 =Exemplary

Please rate the poster/presenter from 0 to 3 on each of the following (circle one):

1. Statement of Research Problem/Rationale: a) Clearly stated questions or hypotheses being addressed 0 1 2 3 b) Well-explained rationale/justification for the study 0 1 2 3 c) Project objectives are clearly outlined 0 1 2 3 2. Literature Review/Background Theory: a) Relevant previous work thoroughly reviewed 0 1 2 3 b) Gap in knowledge/exploration identified 0 1 2 3 c) Succinct 0 1 2 3 d) References are cited appropriately 0 1 2 3 3. Methods (Explanation/Appropriateness): a) Clear description of methods used 0 1 2 3 b) Methods are appropriate to address aim/question 0 1 2 3 4. Analysis/Results: a) Figures/tables used appropriately and clearly to present the data

b) Findings are presented clearly and accurately 0 1 2 3 c) Analysis is well explained and appropriately applied 0 1 2 3

5. Conclusion/Discussion: a) Addressed study’s problem/question 0 1 2 3 b) Conclusions are sufficiently supported by results 0 1 2 3 c) Results are placed into broader framework 0 1 2 3 d) Importance of findings is addressed 0 1 2 3 6. Presentation Overall: a) Effective overall aesthetic/organization of poster 0 1 2 3 b) Flow of information is logical and facilitates understanding 0 1 2 3 c) Presenter summarized study clearly 0 1 2 3 d) Presenter answered questions well 0 1 2 3 e) Length of poster summary (by presenter) was appropriate 0 1 2 3 f) Study is innovative and has potential to contribute to the field 0 1 2 3

Total / 66

Comments:

(Adapted from Florida Atlantic University)

Appendix C

Sample Rubrics | Arts and Humanities

Rubric for Digital Media Project

Exceeding Expectations Meeting Expectations Approaching Expectations

Not Meeting Expectations

Organization Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is clearly and consistently observable.

Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is clearly observable.

Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is intermittently observable.

Organizational pattern (specific introduction and conclusion, sequenced material within the body, and transitions) is not observable.

Content Development

Uses appropriate, relevant, and compelling content to illustrate mastery of the subject.

Uses appropriate, relevant, and compelling content to explore ideas within the context of the assignment.

Uses appropriate and relevant content to develop and explore ideas through most of the work.

Uses appropriate and relevant content to develop simple ideas in some parts of the work.

Storyboard or organizing document

Illustrates the presentation structure with thumbnail sketches of each scene. Notes showing transitions, effects, etc., are clearly planned

Includes presentation structure with thumbnail sketches of each scene. Notes showing transitions, effects, etc. are presented.

Thumbnail sketches are not always clearly marked. Descriptions of scenes, transitions, etc. are incomplete or difficult to follow.

Documents incomplete and unclear.

Technical Elements (audio, lighting, video, timing, etc)

Project is edited with only high quality shots and sounds remaining. Transitions are smooth and project’s timing is appropriate and enhances clarity.

Project is edited with only quality shots and sounds remaining. Pacing and timing are appropriate and enhance clarity.

Project is edited in places. Transitions are noticeable and detract from viewing. Pacing and timing are choppy.

Project is edited poorly with distracting pacing and timing. Clarity is impacted by poor technical control.

(Adapted from Angelo State University)

Appendix D

Sample Rubrics | Science and Math