I originally drafted this as a comment to Claire’s excellent post, but it got long, so I posted it here instead. You should really go read that first.

While the need for education reform is obvious, I think there are two conflicting ideas when you talk about the OECD and PISA and Walter Percy and dogfish. I loved that essay by Percy, and it’s absolutely clear to me that a number of subjects are damaged, in some cases irreparably, by being approached as an object of study. Look at this forum thread: it’s eighteen pages of people slagging many books which are, often rightly, considered some of the best literature has to offer. You might not be able to fix The Fountainhead with a dogfish, but you could certainly do wonders for The Great Gatsby.

What dogfish can’t do is raise PISA scores.

I read over the guidelines and sample questions for the Science section of the PISA. While it’s better than most standardized tests I’ve had the misfortune of encountering during my education, it still runs headfirst into the standard failure modes of the genre. When every question must be asked in a single paragraph, and when the correct answers (of which there are never more than two) must be expressed in a single sentence, the chances for subtlety, poetry and joy must be omitted from the testing pamphlet, if they are even considered.[1] The answers to every question in the exam have already been determined before a single student sits down and unsheathes a #2 pencil. Percy’s dogfish works by surprise; when you see its limpid eyes and flaccid tail where you expected neat rows of type and footnotes you’re forced to engage with it on its own terms, not the ones you carried down the hall from lunch. But there can be no dogfish designed by an international committee: there can be no dogfish under the watchful eye of a certified PISA proctor. The very idea of a dogfish is anathema to standardization and rigorous grading. Dogfish are surprising and PISA, like all standardized tests, awards surprises zero points.[2] There’s no question that an encounter with Percy’s dogfish is a valuable experience that we should like to have in our schools. But, like so many other things that happen while responsible adults are at work, its effect can’t be meaningfully captured by an exam, or at least not one processed by your average scangrade machine.

I also read over the OECD report on how raising our PISA scores will, through the power of increased worker productivity, raise our GDP by 25% over eighty years. Which is pretty excellent, since if there’s anything we can all agree on it’s that we’d all like a little more GDP in our lives. Again, though, I think that Walter Percy and the OECD are looking for rather different things out of our classrooms. There are few professions where being able to quote the Bard on demand is substantively useful, and even fewer where it’s helpful to rattle off a sonnet when confronted with a dogfish. From the perspective of the OECD report[4] those skills are all but invisible since they can’t be measured by the PISA.

Based on my, admittedly personal and possibly idiosyncratic, experience with standardized tests and with the American education system, I suspect that introducing Walter Percy into our classrooms will be extraordinarily difficult. The effects of dogfish can’t be measured with bubble sheets and #2 pencils, and in this day and age that’s going to be a very hard sell. Why should we have our teachers mucking about with the denizens of local rivers when they could be making sure kids know each of the three kinds of word problems that will be on the PSATs? This is a legitimate question. If there’s no test we can administer that shows our dogfish are doing any good, or indeed anything at all, how can we be sure that it’s a worthwhile use of limited classroom time? The answer is that we can’t know, really, not in the same sense that we know it’s good for kids to learn arithmetic and spelling. All we can do is feel in our gut that a life without dogfish is a life extraordinarily beggared, and that making children spend twelve years learning only things they can express in multiple choice is cruel beyond belief.

Fundamentally, I agree that we need to get more interdisciplinary teaching and more Percy-esque learning into our schools. Unfortunately, we aren’t going to be able to justify it with appeals to the PISA.

[1]: This is not to mention the second great failure mode of standardized testing: questions with multiply-interpretable answers. While it’s not as important to the larger discussion that I’m concerned with in this post, it is worth noting that the listed answers to a significant fraction of the PISA science questions, while not wrong per se, greatly restrict the range of the testees thought, if the testee wishes to recieve full marks.

Consider question 3.3, which gives us a table showing the relative effects of different gases on the planet’s greenhouse effect. Carbon dioxide is normalized to 1, while nitrous oxide is 160 and chloroflourocarbons a terrifying 17,000. What other piece(s) of information, the question asks, do we need to know to judge what gas is the main cause of the greenhouse effect? (A) Data about the origin of the four gases, (B) Data about the absorption of the four gases by plants, (C) Data about the size of each of the four types of molecules, (D) Data about the amounts of the four gases in the atmosphere? The only correct answer, according the PISA, is (D). However, a few minutes examination should suggest that we would also want to know (A) and (B). The Earth’s atmosphere is a dynamic system, not a static collection of elements, and without knowing how each gas is produced and consumed, how can we make predictions about their relative frequency tomorrow or ten years from now? And yet that answer, which would demonstrate (at least to me) a higher level of systems and process thinking, recieves no points whatsoever.

[2]: Any attempt to insert a dogfish into a standardized exam would, I suspect, lead to something like the following, which is almost too horrifying to consider:

One day, James enters his English Literature class and discovers a dogfish on his desk. What does he think about the dogfish?[3]

Full Credit
Code 1: Answers that include or reference lines 5 through 8 of Shakespeare’s Sonnet 43.

No Credit
Code 0: Any other responses

[3]: http://www.imdb.com/title/tt0083658/quotes?qt=qt0378241

[4]: Note that I’m not trying to imply that the members of the OECD are heartless corporate stooges trying to turn an entire generation into worker drones for their industrialized masters. The people at the OECD have, I’m sure, only the best of intentions and goodwill towards all. But the OECD report is framed so that the only goal, or at least the only useful, measurable goal, of improving our education system is so that the students emitted by the system can be more productive in their future careers.