01 | We Are More Than a Scantron
Why human intelligence could never be captured by standardized tests.
The timing of major events is often uncanny.
Right at the time our society is reaching the end state of its journey towards replicating human intelligence in artificial systems, we are realizing that there is more to human intelligence than we have accounted for over the past century.
We are simultaneously at the cusp of creating systems that replicate human intelligence as we’ve understood it, and also realizing that our understanding of human intelligence was too narrow.
Let’s start this story with our old friend the Scantron.
I’m 39. While it’s being phased out now, the Scantron was a core part of my education experience. It was likely also for you too. A great invention at the time for high-scale standardized assessment. Typically used to evaluate knowledge recall. What date did this happen? Who invented the whatever? Or the answer to a math problem (have a scratch sheet of paper ready and show your working; but did they ever really check it?).
I don’t have the fondest memories of the Scantron, but I’ll admit it was a good invention for the need of the time.
And that need was to scale high-volume assessment of learning. Reduce the number of hours it would take a teacher to grade. A noble goal. But, as it turns out, only a narrow slice of human capability can be evaluated in this way. Those capabilities being (1) fact retention and recall and (2) analytical reasoning (given some numerical or written information, can you arrive at the right conclusion). What it can’t measure well is creative thinking and innovative problem solving, social and emotional intelligence, or adaptability and decision making under uncertainty. The stuff that actually helps navigate the complexity of the real world.
So it was a tradeoff. Perhaps unknown at the time. Though there is a deeper part of this story in which highly influential people, including John D. Rockefeller, at the turn of the 20th century, intentionally designed education around punctuality, conformity, and rote tasks over independent critical thinking in order to produce “obedient factory workers.” I haven’t been able to verify this concretely, but the general idea and theme of the time does hold up. It was peak industrial era after all.
Either way, as the saying goes, “what gets measured gets managed.” And in education this has shown up as “teaching to the test.” The entire education system began leaning towards the capabilities that were being measured at scale, not because they were the most important capabilities, but because they were the most visible. And so we got really good at knowledge recall (memorizing facts) and analytical thinking (apply logic and formulas to written or numerical information), yet progressively worse at creative thinking, judgment, adaptability, and emotional intelligence.
The Intelligence Quotient (IQ) test then solidified this narrow focus as the crux of “intelligence.” But its original purpose was to identify children who needed additional support to learn the facts and analytical reasoning that were being taught in school at the time. And it all seemed to make sense, especially as developed countries shifted towards the “knowledge” economy and the knowledge-worker become the prestigious line of work. The knowledge-worker being a person who has memorized the facts and procedures of their given field, be it medicine or law or engineering or otherwise.
And so fact recall and analytical thinking became the be-all-end-all of intelligence, and the backbone of the knowledge economy, driving substantial economic productivity. We valued these capabilities so highly that we spent the better part of a century building the machinery to automate them. To automate intelligence. To improve upon it, even. And we’ve seemingly achieved that goal. The two things that AI models already do exceptionally well is information recall and analytical reasoning.
So as far as I can tell, we’ve already achieved the goal of automating human intelligence as we came to see it.
Some people are ecstatic about this. Others are very concerned. Many are somewhere in the middle.
No matter which group you are in, you may find it quite interesting that in parallel to the technological developments (of the neural networks and computer chips and massive datasets) that led to AI over the past few decades, there has been parallel discoveries of equal importance in the social sciences and humanities which are broadening our understanding of human intelligence and what it means to be human (though rediscovering or remembering might be better terms here).
Human intelligence and the conditions that support it.
To keep this article short, I’ll just give a quick overview of what I think is the emerging big picture of human intelligence. This newsletter and the associated podcast, The Alive Conversations, explores this topic from multiple angles and perspectives, so its something we’ll keep returning to. Subscribe below if you’re interested.
As an industrial/organizational psychologist I’ve spent over a decade researching and assessing human capabilities. I’m familiar with the published literature, the research methods, where there’s general consensus versus open questions, and where the blindspots are (there are many). To address the blindspots, I’ve cross trained in machine learning and AI, anthropology, and novel approaches to learning (e.g., equine-assisted learning). In the podcast, I continue to meet people from other disciplines to try and get a holistic and accurate understanding of what human intelligence is and how we can develop it.
I want to share with you my current understanding of the full range of human intelligence, as well as the conditions that support it. Here it is:
Human intelligence is composed of three different, yet overlapping capabilities. For convenience of explanation, each can be described by referring to parts of the body that, for some reason, our language and mental models already have words for. These are useful heuristics only.
The Head. The seat of reason. The head thinks about known information and reasons it through. It’s what we mean when we say to someone “use your head.” It performs well in structured environments where information is known and effective procedures have already been established. This is what we teach in schools and what our economy has rewarded over the past century.
The Gut. The seat of insight and judgment. The gut takes in more information than we are aware of and identifies patterns. It enables us to sit with complex, difficult, uncertain situations and eventually commit to a decision. It’s the insights that come out of nowhere, the sudden creative idea. It’s also the deep intuition and “gut feel” behind confident decision making and judgment calls. The gut develops through experience.
The Heart. The seat of relationship. The heart sees another person clearly and shows up honestly enough to earn their trust. It is the seat of emotional intelligence. It is both the perspective taking and empathy referred to when we say that someone “has a big heart,” yet also the courage and drive to stand up for one’s values that we are referring to when we say “they showed real heart” and “they wear their heart on their sleeve.”
Human intelligence, from my current perspective, is the capability to express all three of these and to be able to move fluidly between them as the situation demands. AI is capable of doing only the first of these three. For now at least. But my hunch is that humans will always excel at Gut and Heart capabilities. So we might be automating Head intelligence, but Gut and Heart intelligence are likely to always stay human. I strongly believe this and just obtained certifications in teaching creative thinking and emotional intelligence because of it.
Crucially, this full range of human intelligence depends on the surrounding environment of the Culture, the Space, and the Work. These categories apply to professional, educational, and also personal life. We’ll explore them more, and their relationship with the three capabilities of human intelligence, in the coming weeks and months.
For now, I’ll leave you with some tips for remembering how to be human in an artificial world.
Today’s tips:
Creativity, gut feel/intuition, or emotional intelligence. Pick one of these topics and spend five minutes reading about it. They are going to become more important very soon.
Educational assessment will begin moving towards evaluating these other components of human intelligence (though it might not call it that). The way this will happen is via portfolios of students’ work. Spend five minutes reading about this if you have kids in school, or listen to episode 21 of The Alive Conversations.
Human judgment is becoming a bottleneck and therefore very valuable. But judgment is built upon our underlying values, which we’ve never been invited to explore. Spend five minutes reading about personal values and how to identify and think about yours. This topic is also discussed in episode 20 of The Alive Conversations.
If you try any of these, let me know what you found and took away by replying to this email.
Now take a real breath, feel the temperature of the air on your face, and notice what its like to be alive in this moment.
Your friend,
James