Breakthroughs rare and decreasing

Innovation is trying stuff. Eventually you find something good enough.

Fellow optimists seem to hold some strange views on how innovation works. This has become apparent when people talk about AI as something that will cure cancer by thinking about it, or derive all of fundamental physics by watching an apple fall.

Statements like these are silly, I want to address some ideas about innovation that might be at the root.

Innovation is mostly trying stuff

The way I think about innovation is Trying Stuff. The best way to determine if something works is to try it in the real world. Most innovation comes from trying different things in a particular domain and keeping what works. Everything else is secondary. Fancy theories make trying stuff more efficient, but aren’t required for trying stuff to work. Standardization and automation are valuable because they accelerate the process of trying stuff.

Most major discoveries were made by people trying stuff, with theory and standardization happening later. Steam engines were in use before thermodynamics, many drugs were discovered on accident, and the green revolution happened because of systematic tinkering.

The implication is that brilliant theory can only get you so far. You need to actually collect data and try things out to make a discovery. The trying things out part is the bottleneck in the process. You have to physically move atoms around and wait for the results. A data center full of Nobel Laureates would spend most of their time waiting for results, not making discoveries¹.

Eventually, trying more stuff is useless

Trying stuff comes at a cost. There is the physical cost of doing the experiments and the opportunity cost that you could have done experiments in some other domain. Combine with diminishing returns and eventually it doesn’t make sense to continue experimenting.

This is stronger than the idea of “low hanging fruit”. Not only do the easy opportunities get found, it stops being useful to even look for new opportunities in a particular area. The marginal benefit is too small.

Most domains see an end of history

At the end of my series on solving climate change I noted:

After looking at the future of cleantech, it feels like we’re nearing an equilibrium. The end of the century may see the end of terrestrial engineering. We found cheap energy sources that won’t be usurped in the forseeable future. The quirks of renewables will propagate into our production processes. After that, what’s left to change?

This is true of all technology. We find an effective way to do something, tweak it a bit, and then kind of stop². This is an odd sentiment in an era defined by ever improving computers, but look outside the enchippening and you’ll see it’s true.

Food products like bread, cheese, meat, beer, and wine are made with techniques that would be recognizable to people centuries ago. Indeed, rain-fed agriculture still provides virtually all of our food, as it has for millennia. It looks unlikely that vertical farming will change that. Of course, we have substantial automation in agriculture, but that’s because we didn’t find an alternative to growing crops. We found new ways to increase yield outside of the growing process.

Even semiconductor fabrication and AI scaling follow the same pattern. We scale up an idea until it reaches physical limits then innovate elsewhere. Smooth progress comes from stacking hundreds of S-curves.

After many S-curves, progress slows. Haber-Bosch is how we produce fertilizer today and I’ve realized we aren’t going to find anything better for the foreseeable future. It takes iron and air and makes bread for gods sake. Or consider that silicon, the second most abundant element in Earth’s crust, has a bandgap that makes it perfect for absorbing the sun’s emission spectrum in solar panels. Need we search further?

Source

Look around at everyday objects and this insight becomes trivial. Things like cups and chairs and pens are already optimized. There is little need for further cup research. There is no magic recipe for making cups more efficiently. Our world is grounded in physics and it’s unlikely we’ll find a substantially better production process.

So innovation jumps around. We encounter a new problem, fiddle around, find a good solution and then move on. Everything converges as we figure out the right way to do things. Unless our circumstances or way of life changes, we should expect society to crystallize around specific technologies and production processes.

That said, there’s still so much room for improvement. The world that we’re converging towards is wildly different from our own. But if we’re going to build a better world, we need a realistic understanding of how innovation works and what is possible.

EDIT: Further Reading

Is Science Slowing Down? By Scott Alexander. Lays out the low hanging fruit theory and suggests it that exponential resource investment to get steady technological gains should be the default assumption.

Tyler Cowen’s book The Great Stagnation.

Are Ideas Getting Harder to Find? Classic paper from Bloom et. al.

EDIT: Growth is Getting Harder to Find, Not Ideas. The fall in firm growth after controlling for idea growth suggests that successive ideas are less valuable, consistent with diminishing returns to more research effort.

Recipes and economic growth: a combinatorial march down an exponential tail by Chad Jones. An economic model of innovation as trying stuff by sampling from a distribution. A toy model that accords with my understanding of the innovation process.

Mundane example of innovation stagnating: why AA batteries haven’t gotten better despite so much progress in battery tech.

There has to be a better way to make titanium. A more technical example of how hard it is to innovate on today’s frontier. Look at how many times people have tried and failed to improve titanium production. Orca has narrowed the possibilities and may, if successful, find the final right way to make titanium.

The lovely book Ignition! gives a firsthand account of scientists studying rocket propellants in the mid 20th century. It tells a story consistent with the model above, people trying everything they could get their hands on, converging to a few right answers, and then moving on. Clark concludes:

There appears to be little left to do in liquid propellant chemistry, and very few important developments to be anticipated. In short, we propellant chemists have worked ourselves out of a job. The heroic age is over.

But it was great fun while it lasted.

Something I wish I’d thought of when writing this piece: consider the innovations of mother nature. Evolution hops around, finding locally available mutations. Species optimize for a particular niche and then stop. When a new niche arises new “innovation” happens until it is filled. Predator and prey find an equilibrium. Consider how invertebrates lack an acquired immune system yet live in peace with their parasites. Human affairs mimic the wild; innovation tinkers, converges, and halts.

EDIT: Or consider sports. Each time a rule is changed, strategies adapt. Sometimes someone finds a new trick or way to game the rules. The league shifts and finds a new equilibrium. There’s not infinite innovation to be found here, just a steady convergence to equilibrium, finding smaller and smaller improvements.

People also pointed me to Ideas aren’t getting harder to find by Karthik Tadepalli. Here was my response:

… We’re saying similar things about low-hanging fruit, but Tadepalli suggests an alternative theory blaming declining allocative efficiency. I’m all for fixing allocative inefficiency, I’m sure that’s part of the problem.

Though I maintain that even if we unblocked the innovation process, that doesn’t change the fact that we’re converging on a right way to produce goods. We’re bound by the laws of physics and economics. …

1. Except in math.

2. The key barrier to further development is often economics. We have found many different ways to make titanium for example, but they simply aren’t competitive with the current system. This is true of many beautiful ideas for how to do things differently.