Setting Long-Term Educational Goals: A Process for State Agencies under ESEA/ESSA, Lecture notes of Business

Download Setting Long-Term Educational Goals: A Process for State Agencies under ESEA/ESSA and more Lecture notes Business in PDF only on Docsity!

Long-Term Goal-Setting

Examples and Development Considerations For Use by State

Educational Agencies in Addressing ESEA Requirements

This series of examples does not necessarily reflect the views or policies of the U.S. Department of Education (Department). The Department has not independently verified the content of these examples and does not guarantee accuracy or completeness. These materials contain the views and recommendations of various subject matter experts as well as hypertext links, contact addresses, and websites to information created and maintained by other public and private organizations. The inclusion of the information in these examples is not intended to reflect a determination by the Department that any activity, product, program, intervention, model, or service mentioned may be supported with Federal funds. The Department has not determined that these examples are effective and does not endorse or recommend any organization, product, or program mentioned in these resources or any views expressed in these examples; the examples described herein are provided merely for informational purposes.

May 2018

Contents

Overview Policy Context A Process for Setting Goals

• Steps for Setting Long-Term Goals

• Building Blocks for Interim Goal-Setting

Examples In Setting Long-Term Goals

• Data Needs and Options

• Categories of Business Rules / Key Decisions

• Example 1a: Varying the Shape of the Trajectory: Linear

• Example 1b: Varying the Shape of the Trajectory: Multiple Stair Steps

• Example 2: Metric is Percent Improvement with Varied Reductions

• Example 3: Confidence Interval-Adjusted Proficiency Target

• Example 4: Recalculated Baseline

Reference Material Acknowledgements and Contact Information

Click on an underlined item to go directly to that section. Slide 2

This presentation defines a process that State educational agencies (SEAs) may use to develop rigorous long-term goals and to determine measurements of interim progress that are aligned to requirements in the Elementary and Secondary Education Act of 1965 (ESEA), as amended by the Every Student Succeeds Act (ESSA).

Examples are included that attend to the following required elements in long-term goal-setting as required in the ESEA:

• Goals are based on academic achievement on statewide assessments

(reading/language arts and mathematics), and high school graduation rate (four-year adjusted cohort graduation rate).

• Goals must be set for the following subgroups of students: economically

disadvantaged students, students from each major racial/ethnic groups, English learners, and children with disabilities.

These examples show how States may use the flexibility in the amended ESEA to vary parameters to set rigorous goals. These examples are not intended to highlight or advocate for specific methodologies and intended to be collectively representative in terms of demographics and geography to ensure relevancy and promote widespread use.

Overview

Purpose and Design of the Examples

Policy Context

ESEA, as amended by ESSA, Section 1111(c)(4)(A)

States must establish long-term goals consistent with the amended ESEA:

• Each State must establish ambitious long-term goals, including measurements

of interim progress toward meeting such goals, for all students and separately for each subgroup of students;

• At a minimum, each State must set goals and measurements of interim

progress for:

 Academic achievement, as measured by proficiency on statewide

reading/language arts and mathematics assessments;

 Graduation rates, including the four-year adjusted cohort graduation

rate (ACGR) and, at the State’s discretion, a more rigorous long-term goal for the extended-year adjusted cohort graduation rate as compared to the four-year; and

 English language proficiency (ELP), as measured by the statewide ELP

assessment.

• The term of the academic achievement and graduation rate goals must be the

same multi-year length of time for all students and for each subgroup of students; and,

• Goals must take into account the improvement necessary for subgroups of

students who are behind on the measures to make significant progress in closing statewide proficiency and graduation rate gaps.

A Process for Setting Goals

Background

The ESEA requires that States establish long-term goals AND measurements of interim progress.

The following section includes:

• A list of the building blocks (the parameters) that States have flexibility in

choosing to vary as they set long-term goals.

• A set of high-level steps that State teams can use to help them work through

the process of setting long-term goals.

• Additional items for State teams to consider as they incorporate goals into the

SEA’s system vision and system framework.

A Process for Setting Goals

Steps for Setting Long-Term Goals

The following steps outline a process for State teams to work through when setting long-term goals.

1. Identify the core need (e.g., to enable more citizens to better financially support their families in the long term). 2. Link the core need to education (e.g., to successfully complete college programs leading to jobs in high demand, high wage areas). 3. Link the core need to K-12 education (e.g., to prepare students to successfully transition to college and career upon graduation from high school). 4. Identify a problem area (e.g., more students need to exit high school better prepared for college/career). 5. Identify particular indicators of success (e.g., high school graduation requirements, college preparedness, college readiness, college success, post-college success). Decide on specific evidence and metrics. Review data and indicators with SEA data team. 6. Describe current performance. Review performance data with SEA data team.

A Process for Setting Goals

Steps for Setting Long-Term Goals (continued)

**7. Describe the long-term goal.

8. Describe how long-term goal can be reached** from current performance in terms of research, past experience, and empirical results. 9. Consider informing long-term goal by empirical results of what has been observed (e.g., top 1% of performance, top 10%, performance outside the state, etc.). Consider business rules necessary for monitoring interim progress. 10. Identify the likely benefits and drawbacks of scenarios of meeting and not meeting the long-term goal. Consider key decisions required to handle special situations. 11. Set long-term goal, balancing previous information. It may require iteratively changing the long-term goal, deeply understanding the empirical data, refining the action plan of how to achieve the long-term goal (e.g., are resources the same, less, or more than devoted to achieving the current performance), and/or checking the consensus and support for the goal and associated plan.

Measures of Interim Progress

After long-term goals have been established, it is possible to establish Measures of Interim Progress (MIPs).

At a high level, the process involves:

• Developing a theory of action for the system that explicitly addresses the goals

and how to achieve the goals;

• Establishing the MIPs using the building blocks (as described in the next

section);

• Developing an evaluation system for the model tied to the theory of action;

and,

• Incorporating results into the accountability system.

A Process for Setting Goals

Building Blocks for Interim Goal-Setting

1. Starting point (baseline)
2. Ending point (long-term goal)
3. Time
4. Shape of trajectory
5. Metric
6. ‘Good Enough’ Criterion
7. Single baseline or recalculated over time

By varying these seven parameters, many different versions of interim goal trajectories can be generated for the same long-term goal, all for the purpose of dealing more appropriately with schools/student groups with different starting points.

The main consideration for choosing between the versions of interim goal trajectories is the amount/rate of improvement viewed as appropriate. A secondary consideration is the desired metric.

Examples of Long-Term Goals

Overview

Setting long-term goals requires the analysis of different data types and setting business rules. These considerations are outlined here and the examples that follow apply the ‘building blocks’ to illustrate different examples of possible long- term goals.

Note that:

• The examples provided are based on percent proficient but the approaches

can be applied to graduation rate as well.

• While most examples only show one of the seven parameters manipulated,

other examples involve variations of multiple factors (e.g., different shape of trajectory and multi-year averaging). States could choose many alternatives as they set goals, and the examples shown are only a small set of the range of alternatives available.

• The targets and time periods for long-term goals shown in the examples are

used only for illustration purposes.

Examples of Long-Term Goals

Data Needs and Options

As SEA teams implement steps 5 and 6 in the process of setting long-term goals, they should engage SEA data teams for support in identifying indicators of success and describing current performance. Gathering the data identified below will facilitate these conversations.

What data are required for the following goal-setting examples?

• Achievement data for the relevant assessments for the current year and prior

years with ability to disaggregate by district, school, and subgroups (or adjusted cohort graduation rates for applicable goals)

• Ancillary data (e.g., partial enrollment information, invalidations,

student/school/district identifiers) to operationalize the business rules for participation, inclusion, and reporting

What additional data may be useful to consider when setting goals?

• Growth data (e.g., VAM scores or growth percentile) with ability to disaggregate

by school and subgroups

• School climate and safety measure

• High school completion and/or drop-out rates

Examples of Long-Term Goals

Business Rules / Key Questions for Consideration

SEA teams can inform steps 8 through 11 in their goal-setting process by clarifying how their goals will be calculated and how progress will be monitored. Working with SEA data teams, the SEA should make decisions and set rules about special cases, appropriate data to use, and minimum requirements for calculations to be carried out, per the categories list below and on the next slide.

Categories of Business Rules and Key Questions for Consideration:

• Participation

 What are the required rates of participation for goal calculation?  What are the rules for determining a participant (e.g., handling partially completed tests)?  What are the consequences of falling below the required rate?

• Inclusion

 What cases are included in calculations (e.g., partial enrollment, irregular administrations, initial vs. retest)?  Is there a minimum required inclusion rate (different from n-size) for reporting or accountability?

Examples of Long-Term Goals

Business Rules / Key Questions for Consideration (cont’d)

Additional Categories of Business Rules and Key Questions for Consideration:

• N-size

 What is the minimum n-size for reporting?  What is the minimum n-size for accountability?

• Missing data

 How to handle incomplete data (e.g., goals are based on mathematics and ELA, but some mathematics tests are missing)?

• Special schools:

 How to handle small schools or those with untested or unusual grade configurations (e.g., K-2 school)?

Examples of Long-Term Goals: Example 1a

Varying the Shape of the Trajectory: Linear

Building blocks/Parameters:

1. Starting point = 2017; 2) Ending point = 2027; 3) Time = 10 years; 4) Shape = Linear; 5) Metric = Percent Proficient; 6) Good Enough Criterion = Met proficient target; 7) Single baseline.

Long-term goal for all students and subgroups of 80% proficient in 10 years. Linear trajectory shown for all subgroups. Subgroups start at different rates of proficiency; the trajectory is more aggressive for subgroups that are further behind.

Examples of Long-Term Goals: Example 1b

Varying the Shape of the Trajectory: Multiple Stair Steps

Building blocks/Parameters:

1. Starting point = 2017; 2) Ending point = 2027; 3) Time = 10 years; 4) Shape = “Stair Step”; 5) Metric = Percent Proficient; 6) Good Enough Criterion = Met proficient target; 7) Single baseline.

Long-term goal for all students and subgroups of 80% proficient in 10 years. “ Stair step” trajectory for interim targets shown. Interim targets are in place for two years. Subgroups start at different rates of proficiency; the trajectory is more aggressive for subgroups that are further behind.

Examples of Long-Term Goals: Example 2

Metric is Percent Improvement with Varied Reductions

Slide 16

Building blocks/Parameters:

1. Starting point = 2017; 2) Ending point = 2027; 3) Time = 2017 or time required given sustained progress; 4) Shape = curvilinear; 5) Metric = status or improvement; 6) Good Enough Criterion = Met proficient or improvement target; 7) Single baseline.

Long-term goal for all students and subgroups is to attain 80% proficient. The interim target is to reduce the % non- proficient by 10% each year or to attain 80% proficient. The interim targets vary for each subgroup.

Examples of Long-Term Goals: Example 3

Confidence Interval-Adjusted Proficiency Target

Building blocks/Parameters:

1. Starting point = 2017; 2) Ending point = 2027; 3) Time = 10 years; 4) Shape = Linear; 5) Metric = Percent Proficient; 6) Good Enough Criterion = Met confidence interval adjusted proficiency target; 7) Single baseline.

Long-term goal for all students and subgroups is to attain 80% proficient in 10 years (subgroups not shown in figure). Linear trajectory. The target is the lower bound of a confidence interval on the target proficiency rate, which accounts for uncertainty associated with lower n-sizes.

Examples of Long-Term Goals: Example 4

Recalculated Baseline

Building blocks/Parameters:

1. Starting point = 2017; 2) Ending point = 2027; 3) Time = 10 years; 4) Shape = Linear; 5) Metric = Percent Proficient; 6) Good Enough Criterion = Met proficient target; 7) Recalculated baseline.

Long-term goal for all students and subgroups is to attain 80% proficient in 10 years (subgroups not shown in figure). Linear trajectory. Each year the actual performance is used to calculate new interim performance targets.

Example 4 cont.

Recalculated Baseline Illustration

Year 1: School starts at 60% proficient and the goal is to get to 80% in 10 years. That school needs to improve by 2% a year for 10 years.

60% 62%^ 64%^ 66%^ 68%^ 70%^ 72%^ 74% 76%^ 78%^ 80%

0%

20%

40%

60%

80%

100%

2016 2018 2020 2022 2024 2026

Initial 10 Year Trajectory

Year 2: School slips to 53% proficient, now the school has 9 years to get to 80% proficient, requiring 3% annual improvement.

53% 56%^ 59%^ 62%^ 65%^ 68% 71%^ 74%^ 77%^ 80%

0%

20%

40%

60%

80%

100%

2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028

9 Year Trajectory

Year 3: School slips to 48% proficient, now the school has 8 years to get to 80% proficient, requiring 4% annual improvement.

48% 52%^ 56%^ 60%^

64% 68%^ 72%^ 76%^

80%

0%

20%

40%

60%

80%

100%

2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028

8 Year Trajectory

Reference Material

Policy Guidance

• U.S. Department of Education ESSA Resources Webpage

• CCSSO ESSA Resources Webpage

Accountability

• U.S. Department of Education’s Office of State Support Technical Assistance

Webpage

• Reeves, D.B. (2000). Accountability in Action: A Blueprint for Learning

Organizations. Englewood, CO: Advanced Learning Press. School Improvement

• Fleischman, S., Scott, C., & Sargrad, S. (2016). Better Evidence, Better

Choices, Better Schools: State Supports for Evidence-Based School Improvement and the Every Student Succeeds Act. Washington, DC: Center for American Progress.

Acknowledgements and Contact Information

These examples were prepared for the U.S. Department of Education's Office of State Support by AEM Corporation with subject matter expertise provided by Brian Gong, Ph.D., Christopher Domaleski, Ph.D., and James Dueck, Ed.D..

If you have questions about how your State can use these examples in development of your long-term goals, contact your U.S. Department of Education, Office of State Support Program Officer using the information below:

Send an email to: OSS.[State] (at) ed.gov, example: [email protected]