(written for a diff target audience than this forum, but with content that's relevant. Mostly me doing a sort of ritualized-public-renunciation of my old ways)

The Hidden and Biased Brain

As a teenager I went on a multi-year pop-psychology reading binge. My best estimate puts it somewhere between 12 and 30 books that all seemed to be talking about the same thing. To describe by pointing, here’s a sample of titles and their taglines:

• Compelling People: The hidden qualities that make us influential
• Future Babble: Why pundits are hedgehogs and foxes know best
• Blink: The power of thinking without thinking
• Predictably Irrational: The hidden forces that shape our decisions
• What Every Body Is Saying: An Ex-FBI Agent’s guide to speed-reading people
• Freakenomics: A rogue economist explores the hidden side of everything
• Fooled By Randomness: The hidden role of chance in life
• Subliminal: How your unconscious mind rules your behavior
• Coercion: Why we listen to what “they” say

And perhaps a bit on the nose:

• You Are Not So Smart: Why you have too many friends on facebook, why your memory is mostly fiction, and 46 other ways you’re deluding yourself.

I’ve got mixed feelings about this cluster of books. On the one hand, it fed and nourished my early curiosity about the mind. On the other hand… a huge amount of it is wrong. If not wrong as in the studies simply do not replicate, wrong in the sense that some interesting studies are used to prop up generalizations about human behavior that are far grander the evidence warrants. I want to explore a very specific and very important thing that this cluster gets wrong, and to do that, we need to explore more exactly what this cluster is.
The unifying pillars of this cluster are The Hidden Brain and The Biased Brain. They form a one-two-punch of a) people don’t actually know what’s going on in their minds and b) what’s going on is largely biased ad-hoc heuristics. The Elephant in the Brain by Robin Hanson and Kevin Simler is the most refined version of the hidden brain thesis. Most books fail to address why we have “hidden parts of our minds”, and why self-deception would even be a thing in the first place. It largely pulls from Trivers on Self-deception, “deceiving ourselves so we can better deceive others”.

Thinking, Fast and Slow by Daniel Kahneman is the most refined version of the biased brain thesis. That has something to do with the fact that Kahneman co founded the entire academic field that gave rise to all of these pop-psychology books in the first place. Back in the 70’s, he and Amos Tversky kicked off the Heuristics and Biases research program. Since I came of age in a world that already had Kahneman and Tversky deep in the water supply, it was useful to get a picture for what the academic scene was like before them. From Kahneman’s book:

“Social scientists in the 1970s broadly accepted two ideas about human nature. First, people are generally rational, and their thinking is normally sound. Second, emotions such as fear, affection, and hatred explain most of the occasions on which people depart from rationality. Our article challenged both assumptions without discussing them directly. We documented systematic errors in the thinking of normal people, and we traced these errors to the design of the machinery of cognition rather than to the corruption of thought by emotion.”

Apparently, there really was a time where academics treated everyone as “rational agents”, where “rational” means a ad-hoc mashup of a mathematical abstraction (VNM axioms of rationality) and a poorly examined set of assumptions about what people care about. Well, I mean people still do that. The real difference now is that if you disagree with them you have a prestigious camp of academics you can point to as on your side, instead of being a lone voice in the wilderness.

(As an aside, I’m not going to put much effort into appreciating the intellectual progress that the Biased Brain and Hidden Brain thesis represent before going into the critique. This is mostly because I grew up with them and the world that came before them is only partially real to me. In my emotional landscape, they are the establishment, so that’s what I’ve gotta speak to.)

So what’s my beef? Where do things go wrong?

As I previously mentioned, there's a shit ton of psychology that simply doesn't replicate. Kahneman makes heavy reference to priming research and ego depletion research, neither of which has anything going for it.

You've also got the problem of defending your usage of the word bias. A bias is a deviation from something, supposedly something that you intended and wanted. In order to talk about bias, you need to have a reference point of what unbiased would mean. I’m not the sort of relativist that thinks this is impossible. It's not. There are things that are good, and there are things that are true. Sometimes it’s easy to find them, sometimes it’s hard. What I do think is that no one that I’ve read on the topic of biases has put in much, if any, effort into outlining a specific stance and defending it. It’s often assumed that we’re all on the same side in regards to what counts as bias.

If you want an extensive academic critique of Kahneman style heuristics and biased research, check out Gigerenzer.

To understand by beef, we need to look more closely at the ways that some of the biased brain books split the mind into parts.

Dual Process Theories

Kahneman's book introduces a framework of two systems to understand human behavior. Everyone more or less takes away that System 1 is fast, biased, and unconscious, while System 2 is slow, rational, and conscious. From the man himself:

“System 1 operates automatically and quickly, with little or no effort and no sense of voluntary control. System 2 allocates attention to the effortful mental activities that demand it, including complex computations. The operations of System 2 are often associated with the subjective experience of agency, choice, and concentration.”

Splitting the mind into two such parts is by no means unique to Kahneman. Kahnemans is just a specific instantiation of a Dual Process theory, which has been getting more and more popular the past half-century. As Kaj Sotala writes:

“The terms System 1 and System 2 were originally coined by the psychologist Keith Stanovich and then popularized by Daniel Kahneman in his book Thinking, Fast and Slow. Stanovich noted that a number of fields within psychology had been developing various kinds of theories distinguishing between fast/intuitive on the one hand and slow/deliberative thinking on the other. Often these fields were not aware of each other. The S1/S2 model was offered as a general version of these specific theories, highlighting features of the two modes of thought that tended to appear in all the theories.”

What these models have in common is that they carve the mind into two processes, one that is intentional, controllable, conscious, and effortful, and another that is unintentional, uncontrollable, unconscious, and efficient. This “bundling” together of properties is the defining feature of a dual process theory, and it’s where everything goes horribly wrong.

The argument is pretty straightforward. Suppose you have four properties that a mental process could have. To simplify, we’ll imagine each property as quite binary, either it’s intentional, or it’s not intentional (this isn’t true, and the gradations of intentionality, consciousness, and effort are incredibly interesting and could be a whole other post). There are 16 different ways you could group together these properties. If you don’t see where the 16 came from, this table should visually explain it:

Dual Process theories start with the idea that the only possible combinations are the very first, and very last ones. That you aren’t going to have any intentional, uncontrollable, unconscious, effortful thought. Or if not impossible, the idea of a dual process theory is that these two combinations are the most likely to occur, and all the others are rare or insignificant enough to be not worth mentioning. If this is not the case, if these properties don’t come all bundled together, then the dual process split fundamentally doesn’t make sense as two categories. The Mythical Number Two, a paper that inspired this post, expands on this point:

“Consider this analogy: we say that there are two types of cars, convertibles and hard-tops. No debate there. But now we say: there are two types of cars, automatic and manual transmission. Yes, those are certainly two different types of cars. And still further: there are two types of cars, gasoline and electric motors. Or: foreign and domestic. The point is that all of these are different types of cars. But we all know that there are not just two types of cars overall: convertibles that all have manual transmission, gasoline engines, and are manufactured overseas; and hard-tops that all have automatic transmission, electric engines, and are made in our own country. All around us we see counterexamples, automobiles that are some other combination of these basic features.”

Unless you see all of the relevant dimensions align together into the clusters you want, there’s not much of a basis for saying “these are the two types of thinking, System 1 and System 2”. You’ve just “arbitrarily” picked two subsets and made them your entire world.

To seal the deal, we need a bunch of concrete counterexamples to the typical dual process bundling. I’m going to save that for a follow up post, though I’m sure that you could come up with some yourself if you brainstormed for 5 minutes. Additionally, you could read the paper I just quoted.


(breaking apart the post here cuz talkyard censors length)
(second half is here)