Priceless. Steve Pinker wrote a spectacular review of Malcolm Gladwell’s latest book, What the Dog Saw and Other Adventures, in the New York Times today. I regularly read and enjoy Gladwell’s essays in the New Yorker, but I find his style sometimes problematic, verging on anti-intellectual, and I’m thrilled to see a scientist of Pinker’s stature calling him out.
Pinker coins a term for the problem with Gladwell’s latest book and his work more generally. Pinker’s term: “The Igon Value Problem” is a clever play on the Eigenvalue Problem in mathematics. You see, Gladwell apparently quotes someone referring to an “igon value.” This is clearly a concept he never dealt with himself even though it is a ubiquitous tool in the statistics and decision science about which Gladwell is frequently so critical. According to Pinker, the Igon Value Problem occurs “when a writer’s education on a topic consists in interviewing an expert,” leading him or her to offering “generalizations that are banal, obtuse or flat wrong.” In other words, the Igon Value Problem is one of dilettantism. Now, this is clearly a constant concern for any science writer, who has the unenviable task of rendering extremely complex and frequently quite technical information down to something that is simultaneously accurate, understandable, and interesting. However, when the bread and butter of one’s work involves criticizing scientific orthodoxy, it seems like one needs to be extremely vigilant to get the scientific orthodoxy right.
Pinker raises the extremely important point that the decisions we make using the formal tools of decision science (and cognate fields) represent solutions to the inevitable trade-offs between information and cost. This cost can take the form of financial cost, time spent on the problem, or computational resources, to name a few. Pinker writes:
Improving the ability of your detection technology to discriminate signals from noise is always a good thing, because it lowers the chance you’ll mistake a target for a distractor or vice versa. But given the technology you have, there is an optimal threshold for a decision, which depends on the relative costs of missing a target and issuing a false alarm. By failing to identify this trade-off, Gladwell bamboozles his readers with pseudoparadoxes about the limitations of pictures and the downside of precise information.
Pinker is particularly critical of an analogy Gladwell draws in one of his essays between predicting the success of future teachers and future professional quarterbacks. Both are difficult decision tasks fraught with uncertainty. Predicting whether an individual will be a quality teacher based on his or her performance on standardized tests or the presence or absence of teaching credentials is an imperfect process just as predicting the success of a quarterback in the N.F.L. based on his performance at the collegiate level. Gladwell argues that anyone with a college degree should be allowed to teach and that the determination of the qualification for the job beyond the college degree should only be made after they have taught. This solution, he argues, is better than the standard practice of credentialing, evaluating, and “going back and looking for better predictors.” You know, science? Pinker doesn’t hold back in his evaluation of this logic:
But this “solution” misses the whole point of assessment, which is not clairvoyance but cost-effectiveness. To hire teachers indiscriminately and judge them on the job is an example of “going back and looking for better predictors”: the first year of a career is being used to predict the remainder. It’s simply the predictor that’s most expensive (in dollars and poorly taught students) along the accuracy- cost trade-off. Nor does the absurdity of this solution for professional athletics (should every college quarterback play in the N.F.L.?) give Gladwell doubts about his misleading analogy between hiring teachers (where the goal is to weed out the bottom 15 percent) and drafting quarterbacks (where the goal is to discover the sliver of a percentage point at the top).
This evaluation is spot-on. As a bit of an aside, the discussion of predicting the quality of prospective quarterbacks also reminds me of one of the great masterpieces of statistical science and the approach described by this paper certainly has a bearing on the types of predictive problems of which Gladwell ruminates. In a 1975 paper, Brad Efron and Carl Morris present a method for predicting 18 major league baseball players’ 1970 season batting average based on their first 45 at-bats. The naïve method for predicting (no doubt, the approach Gladwell’s straw “we” would take) is simply to use the average after the first 45 at-bats. Turns out, there is a better way to solve the problem, in the sense that you can make more precise predictions (though hardly clairvoyant). The method turns on what a Bayesian would call “exchangeability.” Basically, the idea is that being a major league baseball player buys you a certain base prediction for the batting average. So if we combine the averages across the 18 players and with each individual’s average in a weighted manner, we can make a prediction that has less variation in it. A player’s average after a small number of at-bats is a reflection of his abilities but also lots of forces that are out of his control — i.e., are due to chance. Thus, the uncertainty we have in a player’s batting based on this small record is partly due to the inherent variability in his performance but also due to sampling error. By pooling across players, we combine strength and remove some of this sampling error, allowing us to make more precise predictions. This approach is lucidly discussed in great detail in my colleague Simon Jackman‘s new book, draft chapters of which we used when we taught our course on Bayesian statistical methods for the social sciences.
Teacher training and credentialing can be thought of as strategies for ensuring exchangability in teachers, aiding the prediction of teacher performance. I am not an expert, but it seems like we have a long way to go before we can make good predictions about who will become an effective teacher and who will not. This doesn’t mean that we should stop trying.
Janet Maslin, in her review of What the Dog Saw, waxes about Gladwell’s scientific approach to his essays. She writes that the dispassionate tone of his essays “tames visceral events by approaching them scientifically.” I fear that this sentiment, like the statements made in so many Gladwell works, reflects the great gulf between most educated Americans and the realities of scientific practice (we won’t even talk about the gulf between less educated Americans and science). Science is actually a passionate, messy endeavor and sometimes we really do get better by going back and finding better predictors.