RL-as-a-Service will outcompete AGI companies (and that's good)

Hyping up a safer way to develop AI.

Crossposted to LW.

Companies drive AI development today. There's two stories you could tell about the mission of an AI company:

AGI: AI labs will stop at nothing short of Artificial General Intelligence. With enough training and iteration AI will develop a general ability to solve any (feasible) task. We can leverage this general intelligence to solve any problem, including how to make a profit.

Reinforcement Learning-as-a-Service (RLaaS)¹: AI labs have an established process for training language models to attain high performance on clean datasets. By painstakingly creating benchmarks for problems of interest, they can solve any given problem with RL leveraging language models as a general-purpose prior. This is essentially a version of the CAIS model.

Found here. I can’t find the original Epoch article for this.

Both visions are ambitious in the sense that they aim to solve every problem. But RLaaS is more conservative because it tackles each problem separately and relies on some human effort to build datasets. RLaaS requires many models of limited capability honed on specific problems. AGI requires one model with high performance on many tasks.

So, which will dominate the market? I argue that RLaaS has both a better business case and creates less existential risk. It should be promoted.

Why RLaaS will win

RLaaS has proven performance

We already know that training a model on enough task data is sufficient to get high performance on that task. This has been true for the last few decades in machine learning, but has come into focus with language models. AI companies improved model performance across dozens of benchmarks using RL and data from related tasks. The RLaaS model is proven.

The argument that general-purpose reasoning ability will transfer to many domains is more tenuous. Performance gains on e.g. math problems does transfer to other domains, but only in a limited fashion. We've seen dramatic improvements on IMO performance, but this hasn't translated into dramatic gains in other fields.

This is not to say that better reasoning can't produce broad performance gains, just that for a conservative investor today, training a model on well-defined tasks is a safer bet.

RLaaS might cost less

It's safe to assume that a general-purpose model will be more complicated than a specialized model. So inference costs will be higher per task. However, that may be not be a problem if the general-purpose model has much higher performance than the specialized model.²

Something I'm less certain about is the per-task R&D cost. To solve a particular task, building AGI requires substantially more investment than RLaaS, but amortized across enough tasks, the costs may be lower.³

RLaaS is harder to copy

If data scaling drives model performance, you can carve out a safe niche by building your own private dataset for task X. By fine-tuning a model on that data, you now have the best model for completing task X. Competitors would have to go through the same trial-and-error process, which may not be worthwhile if you have a first-mover advantage.

Building AGI, by contrast, is harder to control. For one, there's always the risk that a more capable model is misaligned and simply escapes.

Even with an aligned model, it's not clear that AGI can be kept under wraps. Consider how quickly things like model release dates and algorithms diffuse in the AI industry. If the key is a handful of clever tricks, those details can be leaked pretty easily. And by virtue of being so valuable, there are stronger incentives to steal AGI.

Merely knowing that you created AGI may be enough for others to retrace your steps. It's a lot easier to invent something when you already know it's possible.

AGI is inherently harder to control than a niche dataset.

RLaaS has lower misalignment risk

Present-day models trained on defined tasks are aligned with their users and creators. While these models may be used for malicious purposes, they pose little risk on their own. RLaaS is roughly aligned.⁴

However, the AGI model doesn't have the same assurances. Future AI systems trained under a different paradigm and operating in an open-ended fashion may be misaligned. This adds a substantial downside to developing and using such models.

Conclusion: RLaaS is better and should be promoted

RLaaS has proven performance, lower costs, is more excludable, and is safer. If this holds, most AI companies will pivot away from pursuing AGI and towards RLaaS.

That's good news because it promises a switch to a safer mode of AI development. To the degree that we can promote such a transition, RLaaS should be encouraged. In fact, I'm intentionally using the buzzword "RLaaS" for this reason.

Of course, RLaaS is not without risks; misuse of specialized models is a near term concern. In the future, the concatenation of specialized models may create or assist general intelligences.

But on balance, a transition to the RLaaS model would reduce AI risk and delay the arrival of AGI.

1. This article is the first place I encountered the term.

2. Though during deployment, it may make more sense to train a specialized model on outputs of the general model to save on inference costs.

3. Another possible problem with this story is if AGI can complete tasks that aren't composed of smaller subtasks, unlocking unforseen value that can't be achieved with RLaaS. I'm skeptical, for example, I can't think of a task that can't be completed by organizing enough smart people to work on subproblems. But it's worth mentioning.

4. Models are aligned in practice, but are they aligned in theory? I think we're approaching an understanding of why neural networks generalize both in distribution and out of distribution.

Informally, training chisels cognitive grooves into an agent. Results like the above make me hopeful that prosaic alignment is possible with models trained in the current paradigm.