Florida education commissioner raises concerns about state comparisons with NAEP test
When it comes to comparing academic success across states, there is no better tool than the National Assessment of Educational Progress, better known as NAEP and often called the nation's report card. Fast-rising NAEP scores are one of the main reasons why Florida's ed reforms generated buzz over the past decade.
But now Florida's education commissioner is raising concerns about the validity of state comparisons with NAEP, given big difference from state to state in the percentages of potentially struggling students that are excluded from taking it.
This week - and the timing can't be overlooked - Commissioner Gerard Robinson fired off a letter to David Driscoll, chair of the National Assessment Governing Board that oversees NAEP, and proposed that the board not report NAEP data for states that do not test high percentages of students with disabilities and students learning English. "Because of the importance of NAEP results, it is imperative that NAGB seek ways to ensure all states meet the minimum requirements," Robinson wrote in the letter, dated Tuesday and attached below. "To continue reporting results, despite significant variations in state inclusion rates, calls into question the validity of any conclusions drawn from state-level comparisons to the nation or to other states."
This is wonky as all get out. But interesting on several levels.
The letter comes two months after the most recent NAEP results showed Florida's reading and math gains have largely stalled after a decade of steep increases. It was also sent on the eve of the latest Education Week Quality Counts report, which showed Florida falling from No. 6 to No. 11 among states. Those stagnant NAEP scores are one of two reasons why Florida tumbled.
So, is Florida's concern about NAEP "inclusion rates" sour grapes? Or a valid attempt to find out if the highly regarded national test really is giving us apples-to-apples comparisons? Well ...
The rates, again, do vary wildly. And some of the states ahead of Florida in the rankings do exclude significantly higher numbers of SD and ELL students.
We put together a quick spreadsheet, attached below, that shows the inclusion rates for Ed Week's Top 10 states, for both groups of kids, for all four NAEP tests at issue.
The inclusion goal set by the NAGB is for 85 percent of SD and ELL students selected in the NAEP sample to take the test. Florida is one of the few top-ranked states that clearly meets the goal in every case or nearly every case. Maryland, by contrast - the No. 1 state, according to Ed Week - misses it in nearly every case and in most cases isn't even close. Other states fall short a majority of the time, too.
How much does it matter? Frankly, we don't have the statistical chops to know and we haven't had time to talk to people who do.
After Kathleen Shanahan, the state Board of Education chair, raised the issue with The Gradebook yesterday (in response to the the latest Ed Week rankings), we spoke briefly with Cornelia Orr, the executive director of NAGB and the former top testing official in Florida.
Orr said while the percentages of excluded students in some states may appear large, the raw numbers of students are small, so it doesn't make a large difference in the overall scores. But could it make a small difference? Enough to swing a state's average score a point? Enough to affect a state's rank a notch?
The NAGB has been dealing with this issue for some time. It required the reporting of inclusion rates in the NAEP reports for the first time last year. (You can find the inclusion rates for all states in the 2011 reading report here and the 2011 math report here.)
In his letter, Robinson proposed that the NAGB only report NAEP results for states that meet the minimum standards. He also suggested funding sanctions for states that don't.
(Image from palmbeachpost.com)