Mar 29 2009
Experts–or Not
Once again, the editorial pages of the New York Times have provided me food for thought. And, when I get to thinking, I try to put it on my blog so you all can share it. In this case, it was it was Kristoff’s column on the experts who were apparently caught flat-footed by what, in hindsight, was the inevitable economic meltdown we’re all dealing with right now (some more severely than others). It actually wasn’t really the column that caught my eye so much as the Dr. Fox effect and the findings of a Philip Tetlock of the University of California Berkeley who studied 82,000 predictions by some 284 experts (and described it in his book: Expert Political Judgement ).
The Dr. Fox effect was the way of describing an experiment where an actor was paid to present himself as an expert (Dr. Fox) and give a completely pointless and nonsensical presentation to a group of professional educators. The presentation was presented well (with jokes and likely excellent power point slides, perhaps animation) and left the educators by and large impressed, which either meant that they were too intimidated to admit they didn’t understand or were too amused to protest that it was garbage. Believe me, I’ve seen the Dr. Fox effect in action.
But I found the results of Mr. Tetlock’s study even more fascinating. Apparently, if one examines the predictions of “experts” and compares them to reality, he discovered that their predictions vs. actuality were accurate only slightly better than random. Think about that, not slightly better than, say, the general public, slightly better than “chimps throwing darts at a dartboard.” This was true regardless of the area of expertise, education or years of experience.
In fact, the only differentiation was fame and not how you might expect. The more famous the “expert” the less likely they were to be accurate. The reason Mr. Tetlock decided, was that the media prefers folks that give a definitive answer without codicils or signs conditionals. But I think it’s possible that people feel pressured to give definitive answers when in front of the camera as well.
However, this just reinforces one of my own litmus tests when it comes to scientists (or really an expert): the less open they are to question or adamant that their view is the only view, the less likely they are to really know what they’re talking about. In other words, no real scientist refuses questions or belittles people who disagree. A real scientist understands the limitations of his own knowledge and welcomes anyone who wants to understand. A “bad” scientist thinks that his expertise should quiet all dissent. That’s someone who cares more about appearing right than the truth - which is the antithesis of a real scientist in my opinion. I guess that applies to more than science.
Well, it’s always nice to get data to back my opinion. 
Yeah, I watched Jon Stewart smack down Rick Santelli and CNBC for their reportage on the health of the markets. Especially sickening was to watch all the “experts” make their optimistic forecasts while the market numbers dropped, and dropped, and dropped while the amounts of bailout money rose. And of course to watch CNBC’s sycophantic “reporters” sucking up to the CEOs and not once challenge their (the CEOs’) assurances that there was nothing wrong, everything was just rolling right along smoothly.
I’ll have to get Tetlock’s book. I never heard of the Dr. Fox effect, but it sounds right on the mark. In fact it sounds almost like Jon Stewart’s segment. Great minds surely do think alike!
Too true… too true. In academia it’s the disease of choice, I think. My favorite bogus “expert” is Harold Bloom, who’s published countless books on literature–especially Renaissance lit. And I hate him. I’ve had so many well-meaning instructors and students go horribly wrong because of his ridiculous ideas… and his pomposity is unmatched.
Just a rant from my own sector.
Clearly, I’m not the only one who’s had this experience. Thanks for the feedback.
I’ve been in many a meeting where nonsense was clearly put on screen (often complete with cool photographs and animations, slick twists and unsubstantiated claims, often in complete contradiction to the data they themselves are presenting) - yet I’ll be the only one to challenge it because they are “experts.” It’s nice to know I’m not alone out there.
And, Relax Max, that’s a heck of a compliment. Thanks!
“no real scientist refuses questions or belittles people who disagree.”
I’m with you there! And, yes. It does apply to other areas than just science. “Religion” comes to mind…
Davida
Now I may need to find this book. I’ve noticed the same thing myself.
A few months ago, before the economic collapse, I had started asking people why they were still swallowing the economic claptrap they’d been fed for 3 or more decades, as though we didn’t finally have enough data to judge whether all that laissez-faire theory actually did produce a far better living standard and prosperity for everyone. As far as I could tell, the evidence suggested that it didn’t, and that in fact people weren’t as well off now as they’d been before this stuff started getting put into action.
So I’d say, “Waitwaitwait, you’re accepting what these theorists say, to justify their latest move to privatize stuff or whatever, as though it’s gospel truth, just because they’ve repeated it so often and so loudly. But we’ve got 3 decades of evidence now! Aren’t you even going to check to see if what they’ve claimed 1000 times before has come to pass, before you just accept their word that it will come to pass this time??”
And now I’m saying to people, “Waitwaitwait, you’re listening to the very same people who caused this economic collapse in the first place, and expecting them to tell you how to make it better?? What are you, insane?”
But when you’ve had “experts” proclaiming their theories and expertise loudly and incessantly for several decades, it’s very hard not to just accept that They Must Know What They’re Talking About, Just Ask Them.