How smart can AI get, really?

And can we keep up? (probably not)  
  
͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­͏   ­

| |   
---|---|---  
| | | Forwarded this email? Subscribe here for more  
---  
  
# How smart can AI get, really?

### And can we keep up? (probably not)

| | David Shapiro  
---  
| Nov 29  
---  
|   
---  
   
---  
| | |   
---  
| |   
---  
| |   
---  
| |   
---  
| | READ IN APP  
---  
   
  
I began my YouTube career taking AI "x-risk" very seriously. However, as I engaged with the "safety community" over the years, I realized they were a joke. The vast majority of AI "safetyists" had got their education on LessWrong, which is simply a debate forum. But rather than rehashing my grievances with them, I figured it's high time to ask questions like "how smart will AI get, really?"

| |   
---|---|---  
This meme has been circulating twitter, a take on the "jagged frontier" concept of AI development by Ethan Mollick

For context, this is the original "jagged frontier" blog. In short, this is the idea that machine intelligence surges past some of the things humans can do, while lagging in others. For instance, LLMs are already superhuman coders (beyond the vast majority of coders in certain respects). What the above diagram represents is the idea that, given enough time, "machine capabilities" will become a superset of "human capabilities." We could express that in formal notation as follows:

> `M ⊃ H`

Let M be the set of all machine capabilities and H be the set of all human capabilities. And ⊃ means "superset of" for those who don't know (I had to look it up). 

This is, of course, what the Doomers are afraid of, and what Nobel laureate Geoffrey Hinton, often speaks of. To be fair, Geoffrey Hinton also says that anyone who declares that advanced AI will automatically and inevitably kill us all are equally as crazy as those who say "nothing to see here."

But what does this actually look like? 

Many people engage in "thought stopping" arguments at this point. They usually go something like: _" Well, we can't even agree on a definition of AGI!"_ and the implication is that "therefore, we don't even have to contend with the possibility of M⊃H."

My contention is that we ought to contend with M⊃H. Forget "AGI" and "ASI" for a moment, those are science fiction terms. Wonderful for _Terminator 2_ and fantasy scenarios. But M⊃H is a much more straightforward question: "What if machine capabilities become superset of human capabilities?" It's a much more fruitful conversation. This immediately leads to questions like "how do we characterize machine capabilities vs human capabilities?" Far more interesting than "WhaT's yOur DeFInitION Of AgI"

# Machine Limits

First, it might behoove us to look at, mathematically, what those machine limits might be. What capabilities are machines likely (and unlikely) to gain? If we grant that there is no "secret sauce" to human cognition--that we do not receive hypercosmic signals or rely on quantum chicanery to perform processes--then our brains are (more or less) massively parallel neural networks. 

## Axiom 1--"Bounding Box" of Universal Supersets

Let's first expand our "machines superset humans" concept:

> `Physics ⊃ Math ⊃ M ⊃ H`

`Physics `= "every computation or calculation the universe can do"

`Math `= "every operation that we can define mathematically"

`M `= "every algorithm a machine can actually execute"

`H `= "every process a human can actually do or learn to do"

What this axiom represents is the assertion that "the natural universe (physics) can do things that we can only approximate with symbols and numbers (math), which is itself superset of what we can actually build machines to do as constrained by engineering challenges, and of course all that is superset to what humans can actually do with their brains and bodies."

## Axiom 2--Limited Time Horizons

Due to the limitations of math, physics, and the nature of chaos and complexity, it becomes exponentially/factorially/quadratically more complex to predict further and further into the future. 

Therefore, from a practical standpoint, we can assert "more compute eventually does not equal better predictive power."

Expressed formally:

> `Chaos(System) ⇒ Horizon(System) < ∞`

Where

`System` = some real-world process (weather, markets, etc.)

`Horizon(System)` = furthest time where useful prediction is possible

`Chaos(System)` = the system has sensitive dependence on initial conditions

Put more simply:

"For chaotic systems: More compute ≠ Infinite foresight. There's a hard time horizon where extra intelligence stops helping." (sorry Hari Seldon)

## Axiom 3--The Complexity Wall

In short: "some problems are just hard, full stop." This is P ≠ NP

Let

`Easy(P)` = problem family `P` has solutions that scale "reasonably" with size

`Hard(P)` = problem family `P` blows up (NP-hard, PSPACE, etc.)

`Time_X(P)` = time it takes agent `X` (human or machine) to solve P at realistic scales

Therefore:

> `Hard(P) ⇒ Time_X(P)` explodes for _every_ `X`

"There exist problems P where no agent gets a fast exact solution: smart human, smart AI, alien god, doesn't matter -- the runtime blows up."

In other words, there are problems where more intelligence does not equate to faster or more exact solutions (even if the solution is easy enough to validate, interestingly enough). 

## Axiom 4--Signal Ceiling

You cannot extract more signal from the noise than actually exists. While AI can be better than us, but it can't be better than physics + information.

Therefore:

> `Info_X(Data) ≤ Info(Data)`

Where:

`Data `= observations you get from reality

`Info(Data)` = actual information in the data

`Info_X(Data)` = how much information some agent X can extract from that data

It's fun to imagine that Sherlock can pick up on tiny clues, like a quantum computer, and triangulate "what must be true" from vanishingly tiny clues. You might argue that this is "human intuition" and at a certain point "clues + guess and check + elimination of the impossible" could result in a Sherlockian or Whovian level of wit. At the same time, this is fantasy. Just because you can imagine it, does not make it real, despite what some philosophers would argue. 

## Axiom 5--Bandwidth Gap

Machines have higher throughput than humans. Plain and simple, but it is not infinite. 

> `Bits_H ≪ Bits_M < ∞`

Where

`Bits_H `= total bits of information a human can process

`Bits_M `= total bits of information a machine could process

Humans are far less than machines, and machines are still less than infinite. In other words, there's only so much information a machine can take in, though it is far higher than humans. 

## Axiom 6--Cognitive Horizons

The "cognitive horizon" is the set maximal set of understandings that an agent can possess. This is defined as the set of cognitive primitives (mental artifacts that can be intuitively rendered by an agent) and cognitive operations (manipulations of those artifacts). 

> `Horizon_H ⊂ Horizon_M ⊂ Physics`

Where `Horizon_H `is the cognitive horizon of humans and `Horizon_M `is the horizon of machines. Both are subset to the totality of physics. 

To put it simply "machines can intuitively grasp mental concepts that humans physically cannot." Consider calculus and exponentials. We have no intuition for these because we evolved "locally and geometrically." Whereas machines like AlphaFold can develop intuitions for complex proteins, exponentials, and more. 

# So what? 

Whether you believe in terms like "AGI" or "ASI" is immaterial in the long run. We have machines that can process information in ways that are increasingly alarming insofar as they replicate and supersede the tasks that humans can perform. While many people get hung up on substrate and make grandiose assertions like "AI will _never_ do X" or "humans will _always_ be better at Y" any of those claims require empirical evidence and anyways, neither type of assertion is falsifiable. 

The fact of the matter is that whenever we look at "machine capabilities" versus "human capabilities" through the lens of math and physics, we see that there's not much reason to believe that humans will remain cognitive dominant for very long. Again, some people get hung up on "see, LLM made a stupid mistake!" but that's missing the bigger point. Both human brains and GPUs exploit physics. We're all functionally on the same substrate. 

| |   
---|---|---  
I made this graphic to demonstrate that there is likely a ceiling or plateau of "useful intelligence" but the point is that machines will get far closer to that ceiling than humans ever will. 

The above graphic attempts to summarize the various axioms above. In other words, there are multiple reasons from physics and math that should lead us to believe that there is a "ceiling of useful intelligence" beyond which "more IQ does not yield better outcomes" _however_ , machines are far more likely to approach that ceiling than humans ever will. 

| |   
---|---|---  
Another way to understand this is to compare humans and machines directly. 

You could spend all day teaching the rules of chess to a pigeon, but its cognitive horizon is too small to understand the game, the rules, or even the concept of "winning" (though crows and ravens do!)

From the above axioms, we can anticipate a few possible outcomes:

  1. Machine cognitive primitives will (and likely already have) include primitives beyond what humans can. 

  2. Machine cognition will eventually become ineffable to humans due to our lack of cognitive primitives (like trying to explain chess to a pigeon, the game of chess is ineffable to a pigeon)

  3. One advantage is that reality is often coarse-grained (meaning there's a very real signal in the noise). While machines might better at finding the signal in the noise, we both operate in shared reality. If something does not interact with matter and energy, then it cannot be measured. If it cannot be measured, it is not "real".

  4. And since we share physics (matter, energy, measurement) humans will still likely be able to validate the machine intuition. For instance, a future AI might intuitively grasp quantum mechanics and high energy physics so well that it intuitively comes up with a solution to FTL travel (warp drive) but either way, "can the ship actually travel faster than light" is the real test. 




Now, none of this is to say "machines will automatically be dangerous to humans" my point here is merely to try and articulate and characterize "what do we actually expect machines to do in the long run, based on math and physics?"

You're currently a free subscriber to David Shapiro's Substack. For the full experience, upgrade your subscription.

Upgrade to paid

   
---  
| | | Like  
---  
| | Comment  
---  
| | Restack  
---  
   
  
(C) 2025 David Shapiro  
548 Market Street PMB 72296, San Francisco, CA 94104   
Unsubscribe