ESSA Accountabilty

logo

Bert Stoneberg, Ph.D.
K-12 Research Idaho

ESSA ACCOUNTABILITY - IDAHO

[Current Version] Amended Approved Revised Final Idaho Consolidated State Plan, published June 18, 2019:
http://www.sde.idaho.gov/topics/consolidated-plan/files/Idaho-Consolidated-State-Plan-2019-Amendment.pdf

< ! > Even the 2019 Amended State Plan reports student achievement status and growth using only "percent proficient or above," a metric with limited utility for "trend over time" analyses.

Idaho's accountability plan confuses percentages with percentile ranks.

To qualify for ESSA Idaho must submit a plan for using the funds. A three-member peer review team evaluates the plan and reports to the Secretary of Education whether the plan meets requirements for funding. Idaho has received guidance for the peer review via Peer Review of State Assessment Systems: Non-Regulatory Guidance for States, 2015 (available online at http://www2.ed.gov/policy/else/guide/assessguid15.pdf ). This document will be referred to as PRSAS in the discussion below.

PRSAS ISSUE 1: The State will produce individual student interpretive, descriptive, and diagnostic reports after each administrations of its assessments that provides (1) valid and reliable information regarding the student’s achievement, and (2) information to help parents and teachers interpret the test results and address the specific academic needs of students (pg. 52).

Problem 1. Focusing only on unvalidated student achievement level results is a terrible idea.

Why do our state education officials want to focus all of our understanding of student achievement in Idaho on TRIAL DATA (i.e., on the percent of students "at or above proficient)? There is NO evidence to support claims that achievement level percentages are “reasonable, valid and informative to the public.”

The SBAC/ISAT developers set the achievement level percentages to mirror closely the NAEP achievement level results, so interpretations that apply to NAEP achievement level results also apply to the SBAC/ISAT achievement levels.

Excerpt from the National Center of Education Statistics website: The 2001 NAEP reauthorization law requires that NAEP achievement levels be used ONLY on a trial basis until the Commissioner of Education Statistics determines that the achievement levels are “reasonable, valid, and informative to the public.” So far (i.e., as of July 7, 2017), no Commissioner has made such a determination, and the achievement levels remain in a trial status.

PRSAS ISSUE 2: State reports will show student achievement in terms of the state's grade-level academic achievement standards (pg. 52). This could be demonstrated by a study finding that achievement scores at each tested grade correlate positively with teacher judgments about student readiness for the next grade level (pg. 38).

Problem 2. The SBAC's "percent proficient and above" statistic does not match Idaho's grade-level academic achievement standards.

Idaho's ESSA plan states that statewide SBAC/ISAT scores in Mathematics and English Language Arts used for academic indicators (i.e., percent proficient and above) have met validity and reliability criteria required by FRSAS. However, this is not the case. SBAC "proficient" is not the same thing as "proficiency in the subject."

Under NCLB, there were two federal definitions for “proficient,” the NAEP definition and the NCLB definition. The NAEP definition for “proficient” was equivalent to a classroom grade of A. The definition for NCLB “proficient” included the grades C and B, to represent grade-level expectations. (It is noteworthy that the definition for NAEP "basic" included the grades C and B, to represent "proficiency in the subject" as used in everyday language understood by parents and the public.)

According to FRSAS, the peer review team’s task is to verify that Idaho’s “proficient” achievement level representes an attainment of grade-level expectations sufficient to qualify a student for the next grade level. In the previous ISAT, implemented for NCLB, the state “proficient” achievement level did represent attainment of the state’s grade-level expectations.

The SBAC developers caused the ISAT achievement levels to mirror the NAEP achievement levels as closely as possible. Thus, interpretations that apply to NAEP achievement level results now apply to the SBAC achievement levels. Since NAEP proficient was akin to an A, the SBAC proficient is now akin to an A. I doubt seriously that few, if any Idaho teachers would agree that an A performance on the SBAC (a "proficient" score) is a must to consider a student to be ready for the next grade level.

SBAC/ISAT's "basic" achievement level represents Idaho's grade-level expectations because SBAC "basic" has the same meaning as NAEP "basic," which includes students who are "proficient in the subject," using the common language meaning for the term. If "achievement level percentages" MUST be used, at the very least the SBAC/ISAT "proficient and above" score should be replaced by the SBAC/ISAT "basic and above" score!

PRSAS ISSUE 3. The state’s reporting system facilitates timely, appropriate, credible, and defensible interpretation and use of its assessment results (pg. 51) .

Problem 3. The state's reporting system must be based on SBAC/ISAT results other than achievement levels scores, other than "percent proficient."

The SBAC developers used the Item Response Theory (IRT) model. The IRT model creates scale scores for a test; it does not create achievement level scores. Since achievement level scores have never been shown to be “reasonable, valid and informative to the public,” they do not meet the "defensible interpretation and use" guideline. SBAC scale scores and their standard score transformations, however, can meet the "defensible interpretation and use" guideline.

Achievement levels scores can sometimes lead to a false interpretation of student achievement. Percentiles are a standard score transformation of scale scores that have a long history of use for reporting student achievement. A two-page snapshot report illustrates this problem by comparing the statewide "percent proficient" results with the corresponding percentile results on the 2015 and 2016 SBAC/ISAT 6th grade English Language Arts tests:

Gain in achievement level results. The State Department of Education released the statewide SBAC “percent proficient” statistics for 2015 and 2016, for each grade-subject. The percent proficient chart for sixth grade English Language Arts shows that 49% of Idaho’s sixth graders scored “proficient” in 2015, which rose to 51% in 2016.

But loss in percentile results. The chart for five percentiles (derived from the scale score n’s, means and standard deviations) shows that Idaho sixth graders across the board from high scoring student (90th percentile) to low scoring student (10th percentile) in 2016 scored lower in English Language Arts that their counterparts in 2015.

A 24-page report (mostly page-consuming tables and graphs) looks at all of the SBAC/ISAT 2015 and 2016 percentile results for grades 3-8 and 10, for both English Language Arts and Mathematics.

Learning Point Associates. (2009, August). Connecting Research to Practice: Knowing Who Is Proficient Isn't Always Sufficient.
Retrieved July 5, 2017, from http://files.eric.ed.gov/fulltext/ED509965.pdf

LEARN ABOUT THE "ONCE AND FUTURE" MEASURES OF SCHOOL PERFORMANCE

1. AVERAGE SCALE SCORE. (Statistical difference between two means.)

2. PERCENTILE. Percentiles give us performance information across the whole distribution of student scale scores, not just students getting A’s. They let us compare students performing at five different percentiles. The five percentiles (i.e., scale scores) are:
- 90th percentile: High Score (above 90% of the student scale scores)
- 75th percentile: High Average Score (above 75% of the scale scores)
- 50th percentile: Average Score for Grade Level (above 50% of the scores)
- 25th percentile: Low Average Score (above 25% of the scale scores)
- 10th percentile: Low Score (above 10% of the student scale scores)
The 25th to the 75th percentiles include the middle half of students who took the test, and the 50th percentile is the middle half of students.

<> This 24-page report (mostly page-consuming tables and graphs) looks at the SBAC/ISAT 2015 and 2016 percentile results [ PDF ].
<> This two-page snapshot titled SBAC/ISAT “proficient achievement level vs. percentiles” gives completely different pictures of student achievement in Idaho [ PDF ].
<> This document of tables lets you find the percentile rank for each 2016 ISAT scale score, whether individual score or group average score. [ PDF ].

3. SIMPLE EFFECT SIZE. (Practical difference between two means.)

Comments submitted regarding "Idaho Accountability System Network"

<> May 5, 2016 - Idaho K-12 Accountability System Network, Comment #1 [ PDF ].
<> June 6, 2016 - Idaho K-12 Accountability System Network, Comment #2 [ PDF ].

Disclosure: K-12 Research Idaho is a personal public service project funded from retirement benefits.
Bert Stoneberg was the NAEP State Coordinator for Idaho from 2002 to 2012.

Go to: K-12 Home Page