Blogging Disrupting Class #11: Chapter 7 -- Improving Education Research (Part 2)

The previous post summarizes how The first part of Chapter 7 discusses some ideas for improving education research, more specifically improving descriptive research and evolving toward prescriptive research as a superior form of research. I was on board for the first part, but the transition from descriptive to prescriptive bodies of understanding (pp. 167-170) is where I jump off.

Why? Maybe Note #9 explains it: the authors' guiding assumption is that "the value of a theory is assessed by its predictive power" (p.176) -- in other words, a purely rational-scientific approach to determining "truth", even if a body of understanding only "asymptotically approaches truth" (i.e., you can get ever closer, but never quite get there). But I'm not even sure I object to the method being proposed (diagram, p.169), but rather the purposes to which it is being put. In other words, I don't see why their methods have to be described as prescriptive or as yielding prescriptive results. Their culminating example illustrates this: they say that descriptive research might conclude something like 'phonics produces better results on average', while prescriptive research would yield something like 'phonics produces better results with Student x who's strong in Intelligence A, while Whole Language produces better results with Student y who's strong in Intelligence B (p.170).

HUH? This is not individualizing education for students; it's standardizing education based on an additional construct -- the authors seem to believe that 'intelligence types' are the Great Customizer. But what if Whole Language works better with Student z who's strong in Intelligence A? Or better yet, how about if we customize the use of different combinations of phonics and whole language for each individual student based on their particular needs, and just bypass altogether the whole silly drive to prescribe? So this example illustrates for me precisely why we don't want to be prescribing at this point.

But it's the next example IMO which fully reveals the limitations of the authors' conceptual approach to education reform (and by extension education research). The history of manned flight anecdote (pp.170-172) is an excellent example of how scientific inquiry works, but it's not a particularly good example of how education works. This is a really important point, so I'm taking some extra time here to try to figure out how to explain this.

Discovering the principles of manned flight was a process culminating in the discovery of a single solution -- there are now many variations on this solution, but the outcome is unitary in some fundamental way, i.e., there is a "causal mechanism" to be discovered (p.171). But education is not a process of discovering a single "causal mechanism"; it involves multiple outcomes, each of which is on some level as unique as each individual.

Partly this is a semantic issue -- there are certain "desired outcomes" (p.172) we all want -- having all children able to read, for example. But this is not analogous to learning to fly; there is not a single unitary principle or causal mechanism which causes children to read, or learn. So it's asking for trouble when we try to oversimplify reality to arrive at prescriptive solutions, as in the phonics vs. Whole Language example (p. 170). Ironically, looking for prescriptive solutions seems to lead inexorably to standardized solutions rather than individualized ones, a trap which even these authors have failed to avoid despite their best efforts.

OK, moving on -- the next paragraph includes this sentence, noted approvingly: "Many federal K-12 grant programs now demand research that is based on randomized controlled trials" (p.172). Those of you familiar with my work on the comparison trap know that this is a pet peeve of mine. Isn't randomizing the very antithesis of customizing? Let alone the ethical issues involved -- even medical research suspends the use of randomized controlled trials if they think they've discovered a treatment which would benefit all subjects. And somehow they apparently think this is compatible with the next source they cite approvingly, Meredith Honig, who argues that "the essential implementation question then becomes...what is implementable and what works for whom, where, when, and why?" At least this latter formulation offers some compatibility with customizing learning.

The next section is even more condenscending -- a description of the differences between reliability, internal validity, and external validity (p.173), which they present as if it were revealed knowledge unknown to educators. Hey guys, next time open up an education research textbook first, OK? Maybe try a little Vygotskian promixal zone of development strategy? Linking what the readership already knows to something they might not yet know?
They get close when talking about external validity (p.173-174) -- but again, ultimately fall short. They seem to believe that "getting the categories right" is a relatively straightforward process which results in a construct of reliable and fixed durability. But in reality, there are good reasons why educators and stakeholders argue endlessly about categories -- more on that in a later post.

And so they conclude the chapter by scolding practitioners for enacting reforms without defining the "categories or situations in which the recommended actions would be effective" (p.174). I re-read this section in vain for some indication of how this connects to customization, but it sounds like simply a recommended strategy for achieving standardized results.

I'm not entirely comfortable with the harshness of my critique of this chapter. In some ways, they are criticizing the same things I've criticized about much education research, especially comparative studies. Discovering anomalies is potentially a very useful strategies. But they've put their approach into a rational-scientific framework espousing experimental approaches as if the truth is out there, waiting to be attained (even if only asymptotically). Thus Note #1 (p.174) approvingly cites a National Academy of Sciences study which trashes educational research -- fine, so long as one acknowledges that "randomized, controlled studies" are not the antidote. And yes, take a systemic approach which at least references cultural and other underlying factors rather than focusing too narrowly on pedagogical and curricular factors (also from Note #1).

But consider this quote from Note #13 (p.177): "Teachers and administrators need to know reliably what will happen if they implement a policy in their specific situation, not what will happen on average." In a world where future problems are not even known yet and multiple solutions are the order of the day, are "reliable results" really what education is about? Or is it better to be thinking about designing a system which produces unexpected, new results, even if these results are not necessarily "reliable"? I'd like to think that the issue is only a semantic one -- for instance, a shared common goal on being able to reliably produce unexpected results -- but I suspect that the real issue is deeper than semantic: that Disrupting Class proposes a solution which supports customized inputs, but standardized results.

Chapter 7 (Part 1)
Chapter 6
Chapter 5
Chapter 4 (Part 3)
Chapter 4 (Part 2)
Chapter 4 (Part 1)
Chapter 3
Chapter 2
Chapter 1
Introduction