Tuesday, June 19, 2018
Opinion

For-profit standardized testing industry can't be trusted

The scores for the writing portion of this year's FCAT plummeted so precipitously that the abilities of Florida's student writers aren't even being called into question. The validity of the scoring statistics are. While I don't want to say "I told you so" regarding the dubiousness of those statistics, I did tell you so, as my 2009 book highlighted in detail all the ways the numbers produced by the for-profit standardized testing industry cannot be trusted.

Take the stats produced at Pearson scoring centers around the country, where I worked for the better part of 15 years. On the first project I worked scoring student essays, I had to pass a qualifying exam to stay on the job. When I failed that qualifying exam (twice), I was unceremoniously fired. So were half the original hundred scorers who had also failed the tests. Of course, when Pearson realized the next morning they no longer had enough scorers to complete the project on time, they simply lowered the "passing" grade on the qualifying test and put us flunkies right back on the job.

Yes, those of us considered unable to score student essays 12 hours before were welcomed back into the scoring center with open arms, deemed qualified after all.

Such duplicity was not an aberration in my experience either. For a decade and a half I saw every sort of corporate chicanery and statistical tomfoolery. The test-scoring industry seemed focused on getting deadlines met, projects completed and scores put on tests, but only then did any thought seem to be given to meaningful scores being put on them.

I regularly saw unqualified people (myself included, apparently) keep their jobs scoring student responses even when they were altogether no good at the job, either when the acceptable qualifying grades were dropped so low that anyone could meet them, or when the correct answers to the qualifying exams were handed out even before the tests were taken.

I regularly saw statistics get doctored to make group reliability numbers (agreement between the scorers) look better than they really were, as high reliability stats were necessary to convince customers how standardized a job was being done and how "valid" the work really was. I regularly saw distribution numbers fixed to make score results look however a client might have wanted.

Once I attended a range-finding meeting with other test-scoring experts and English professors from around the country, the bunch of us trying to figure out how to score writing samples for a national test. After that group of experienced test scorers and esteemed writing teachers had hammered out some consensus regarding the writing rubric and writing samples we'd been reviewing, we were told we were scoring "wrong." We test-scoring experts and writing teachers were told our scoring wasn't matching the predictions of the omniscient psychometricians (statisticians/testing gurus), and we were told we had to match those predictions even though the pyschometricians had never actually seen the student responses.

When the next year I read in the New York Times that student writing scores had ended up exactly in the middle of the psychometricians' predictions, I can't say I was surprised: We had made sure they did.

And that's the thing: In my experience, the for-profit test-scoring industry could produce results on demand. There was no statistic that couldn't be doctored, no number that couldn't be fudged, no figure that couldn't be bent to our collective will. Once, when a state Department of Education (it wasn't Florida's) didn't like the distribution of essay scores we'd been producing over the first two weeks of a project, we simply followed its instruction to give more upper level scores. "More 3's!" became our battle cry on that project, even if randomly giving more 3's was fundamentally unfair to all the students whose essays had been assessed differently in the days before.

In the end, I guess I'm saying you probably needn't worry too much about this year's falling FCAT scores, because they're only a number. If you want a different number next year, just ask; surely Pearson will just make more.

Todd Farley is the author of "Making the Grades: My Misadventures in the Standardized Testing Industry." His opinions have been published in the New York Times, Washington Post and Education Week.

Comments
Editorial: A court victory for protecting Florida’s environment

Editorial: A court victory for protecting Florida’s environment

A Tallahassee judge has affirmed the overwhelming intent of Florida voters by ruling that state lawmakers have failed to comply with a constitutional amendment that is supposed to provide a specific pot of money to buy and preserve endangered lands. ...
Updated: 10 hours ago
Editorial: Trump should stop taking children away from parents at the border

Editorial: Trump should stop taking children away from parents at the border

Innocent children should not be used as political pawns. That is exactly what the Trump administration is doing by cruelly prying young children away from their parents as these desperate families cross the Mexican border in search of a safer, better...
Published: 06/17/18
Updated: 06/19/18

Editorial: ATF should get tougher on gun dealers who violate the law

Gun dealers who break the law by turning a blind eye to federal licensing rules are as dangerous to society as people who have no right to a possess a firearm in the first place. Yet a recent report shows that the federal agency responsible for polic...
Published: 06/17/18
Updated: 06/18/18
Editorial: Encouraging private citizens to step up on transit

Editorial: Encouraging private citizens to step up on transit

The new grass-roots effort to put a transportation package before Hillsborough County voters in November faces a tough slog. Voters rejected a similar effort in 2010, and another in 2016 by elected officials never made it from the gate. But the lates...
Published: 06/15/18
Editorial: 40 years later, honoring remarkable legacy of Nelson Poynter

Editorial: 40 years later, honoring remarkable legacy of Nelson Poynter

Forty years ago today, Nelson Poynter died. He was the last individual to own this newspaper, and to keep the Times connected to this community, he did something remarkable. He gave it away.In his last years, Mr. Poynter recognized that sooner or lat...
Published: 06/15/18

There was no FBI anti-Trump conspiracy

The Justice Department released Thursday the highly anticipated report on the FBI’s handling of the Hillary Clinton email probe and other sensitive issues in the 2016 election. It is not the report President Donald Trump wanted. But there is enough i...
Published: 06/14/18
Updated: 06/15/18

Voter purge may be legal, but it’s also suppression

The Supreme Court’s ruling last Monday to allow Ohio’s purging of its voter rolls is difficult to dispute legally. While federal law prohibits removing citizens from voter rolls simply because they haven’t voted, Ohio’s purge is slightly different. T...
Published: 06/14/18
Updated: 06/15/18

Editorial: Free rides will serve as a test of whether the streetcar is serious transportation

Who wouldn’t jump at the chance to ride for free?This fall, the TECO Streetcar Line eliminates its $2.50-a-ride-fare, providing the best opportunity yet to see whether the system’s vintage streetcar replicas can serve as a legitimate transportation a...
Published: 06/14/18
Updated: 06/15/18

AT&T and the case for digital innovation

A good way to guarantee you’ll be wrong about something is to predict the future of technology. As in, "One day, we’ll all …" Experts can hazard guesses about artificial intelligence, driverless cars or the death of cable television, but technologica...
Published: 06/14/18
Editorial: State, nonprofits share obligation to help Hillsborough’s foster kids

Editorial: State, nonprofits share obligation to help Hillsborough’s foster kids

The Florida Department of Children and Families has correctly set a quick deadline for Hillsborough County’s main child welfare provider to correct its foster care program. For too long the same story has played out, where troubled teens who need fos...
Published: 06/14/18