Reality Check

Wyoming's Joint Interim Education Committee and Select Committee on Education Accountability both met on November 14^th and 15^th. The Legislative Service Office opened the agenda on the 14^th with a fiscal update including a forecast out to 2021-22 which was grim, to say the least. After that reminder, the committees continued conversations on topics from their respective prior meetings.

Two individuals, one of whom is heavily relied upon for advice about education accountability, made unsupported assertions to the committees that Wyoming has improving National Assessment for Educational Progress (NAEP) test scores, in particular those of disadvantaged students where improvement was described as "dramatic". The intent behind the statements seemed to be to caution the committees about looming spending cuts and the perceived risk of losing those gains, implying spending and outcomes are directly linked.

This article is written to examine these statements and see how true they are.

For that purpose, average test scores from the NAEP website were downloaded for 4^th and 8^th grades in reading and math for two populations, all students and students eligible for the National School Lunch Program^[1]. Eligibility for school lunches is used as a proxy for disadvantaged students because NAEP does not provide data for any group defined as "disadvantaged." Both populations were then split into national and Wyoming sub-groups.

We look at the data in two ways. First, we looked at the raw percentage change in average scores from the earliest date available through 2015 in order to see the simple overall change from beginning to end. Second, we looked at the percentage change in average scores year by year over the same periods. The purpose of this is to see the size of change year by year and the trend. At the end of this article, we'll put those statistics in context with the changes which have occurred in state spending.

Graphs 1 through 4 show the average Grade 4 and Grade 8 reading and math scores by test year for all

students nationally and for all students in Wyoming. They also show the percentage changes from 1998 (or 1996) to 2015. For example, nationally the percentage changes in Grade 4 reading were 3.6 and 0.7 percent, respectively. Overall, in all cases, Wyoming outperformed the national result, math improvement was better than in reading and improvement dissipates in Grade 8. The point, however in our view, is in no case does the percentage change over the 17 to 19 year periods constitute dramatic or impressive improvement.

In graphs 5 through 8 we apply the same analysis to data for the sub-populations eligible for the National School Lunch Program. "The National School Lunch Program is a federally assisted meal program operating in public and nonprofit private schools and residential child care institutions. It provides nutritionally balanced, low-cost or free lunches to children each school day. The program was established

under the National School Lunch Act, signed by President Harry Truman in 1946." Each year the Department announces adjustments to the Income Eligibility Guidelines to be used in determining eligibility for free and reduced price meals and free milk. "The guidelines are intended to direct benefits to those children most in need..."

In all cases, the percentage improvement is better in the School Lunch groups than in the all students groups. However, Wyoming lags the national result in both Grades 4 and 8 for reading and Grade 8 math. And, again, it's unlikely everyone will agree the improvement is impressive given the length of time the change took to occur.

Now let's examine the timing of the changes. Graph 9 shows us Grade 4 reading, all students, where yearly changes have averaged 0.5 percent. Wyoming's trend is slightly up. Nationally, the trend is slightly down. (Note: no result is calculable for Wyoming in 2000 and 2002.)

Graph 10 shows us Grade 8 reading, all students. Wyoming's progress has been irregular and averaged 0.3%. Overall, the trend is down, while nationally it is slightly up.

Graph 11 reports Grade 4 reading for the School Lunch Program. Wyoming averaged 0.6 percent per year and its trend is down. The national trend is also down.

Graph 12 illustrates Grade 8 reading for the School Lunch Program. Wyoming averaged 0.3 percent per year and its trend is down. The national trend is flat.

Graphs 13 through 16 show math: all have the same pattern for Grades 4 and 8 and for both all students and the School Lunch Program. Significant percentage changes occurred in 2003 in all categories and the amount of change has decreased every year since. Interestingly, the national figures show very similar patterns.

By way of contrast, Graph 17 describes the history of percentage changes in operating revenues per student per year. Annual increases as large as 12, 16 and 23 percent occurred in school years 2001-02 through 2006-07. Overall, change averaged 5.9% and, cumulatively, the changes shown amount to 105 percent - more than double. Graph 18 shows it in dollar terms.

Reasonable people can differ when interpreting data so it's important for them to actually see the data and reach their own conclusions. Our opinion is that while progress in NAEP scores is being made, it is modest, at best. In addition, as has been said before, there appears to be no correlation between the enormous increases in spending and changes in test scores. For that reason, there is little risk scores will decrease if funding decreases.

Testimony to our legislative committees should be supported by data and should accurately reflect that data. We hope the foregoing is useful for the ongoing discussions in Wyoming about assessment, accountability and funding.

^[1] Readers should note that NAEP cautions users of their data against drawing conclusions about very small populations. Wyoming's School Lunch Program population would likely be considered very small.