Links #28

Chase's annual opinions on solar, an AI deluge, and more.

1.

Jenny Chase posted her annual “opinions about solar” thread on Bluesky. See here for a formatted version. Some highlights:

4. We don’t need a solar technology breakthrough. The challenges to building solar are usually getting a grid connection and planning permission, or increasingly, power price cannibalization. Of which more later in thread.

14. Low power prices may be great for consumers but they are very bad if you’re trying to build more clean power plants. Without demand-side flexibility measures, the energy transition will fail before fully pushing fossil fuel out of the mix. Which is what we must do.

15. It’s very easy to say “but batteries!” and those are definitely part of the solution. California has over 14GW of batteries in a grid with roughly 50GW peak demand, and the reliability of the grid has improved as its carbon emissions go down.

16. ...but batteries are still small. In 2024, about 181GWh of lithium-ion stationary storage was deployed worldwide, plus 974GWh lithium-ion batteries in vehicles. (www.bnef.com).

36. We’re finally getting serious about net zero carbon. Getting that last 5-20% of carbon out of power will be hard, and require some expensive solutions. The first 80-95% is easy-ish but we’re getting on with it.

51. For 6 years I have been refusing to get excited about perovskites until a perovskite company can disclose a commercial partnership with a named major module manufacturer. They have now. Still not excited. Crystalline silicon is honestly pretty great.

Much more in the thread!

2.

I have a deluge of AI links.

LoRA

Low-rank adaptation (LoRA) is an efficient way to fine-tune a language model for a task. Some recent news on this technique:

Learning without training: The implicit dynamics of in-context learning. In-context learning is just giving the model more background information in the prompt. Turns out this extra context is essentially applying a LoRA to the MLP layers of the language model/

Empirically, LoRA Without Regret shows that LoRA outperforms other fine-tuning methods.

Imagine indexing the internet like this. Each page has an LLM-accessible “summary” that’s just a LoRA a model can add to its weights. LLM’s could stream the internet much faster that way¹.

LoRA fine-tunes can be quite practical to serve at scale. Two demonstrations along these lines:

Punica: Multi-Tenant LoRA Serving

S-LoRA: Serving Thousands of Concurrent LoRA Adapters

Deeper understandings of neural networks

Making sense of parameter-space decomposition. Understanding neural networks as composing what are essentially LoRA’s.

Understanding Optimization in Deep Learning with Central Flows. How stochastic gradient descent automatically avoids regions with high curvature. These regions should in theory make SGD unstable, but their theory shows why it works.

Sparse Networks and Lottery Winners good intuition and toy models showing how neural networks find a small sub-network that solves your problem.

Speedrunning and the singularity

Donoho’s Frictionless Reproducibility is my favorite vision for the future of science. A few weeks ago I realized that NanoGPT speedrunning is a great way to accelerate open innovation in language models:

How the NanoGPT Speedrun WR dropped by 20% in 3 months gives us a detailed look at the progress on this benchmark. It’s really happening!

Andrej Karpathy (the creator of nanoGPT) has extended this idea with nanoChat, training “… the best ChatGPT that $100 can buy”.

Assorted

Cartridges: Storing long contexts in tiny caches with self-study. When a language model absorbs a lot of text, the semantic information from that text gets stored in what’s called a KV cache. This gets referenced as the model produces a response. The new work figures out how to compress the KV cache by 38x by essentially “training” the bits stored in the KV cache to optimize for information retrieval. I wonder if LoRA’s could be stored like this too².

Evolution Strategies is interesting as a “third way” between genetic algorithms and gradient descent training. Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning uses it to train a language model. It would be neat to see ES make a comeback, though I don’t expect much.

Everything else

Hyperstimuli are Understimulating. Addictive stuff leaves you wanting more by being unsatisfying.

I take antidepressants. You’re welcome. Perhaps anti-depressants could cure misanthropy. Should we all be on them?

Ava is writing a series on friendship, looks interesting.

Family Conflict, Humanism and Formalism a good way to think about how people approach relationships. “Formalists view relationships through rules and obligations … Humanists view prioritizes emotional outcomes … Neither approach is clearly superior …”

Reasons and Persons: Watch theories eat themselves. An accessible summary of Parfit’s book.

Study: Giving cash to mothers cut infant deaths in half. That’s very good news!

America could have $4 lunch bowls like Japan—but our zoning laws make them illegal. Let them cook!

Rivers are now battlefields. How desalination tech could help with national security and deter aggression.

Incentive-Compatible Recovery from Manipulated Signals, with Applications to Decentralized Physical Infrastructure

Cryptocurrencies rely on the internet to function. But what if you didn’t want to trust even our communications infrastructure? Kryptoradio is a defunct project that allows people to observe the blockchain over the radio, completely off-grid. In theory you could run a worldwide cryptocurrency over radio alone.

Transverse Electron Beam Shaping with Light. “We can realize both convex and concave electron lenses with a focal length of a few millimeters, comparable to those in state-of-the-art electron microscopes.” Pretty interesting because electron microscopes can be used to make computer chips and image a bunch of stuff. Could this lead to cheaper/better electron microscopes?

Combinatorial protein barcodes enable self-correcting neuron tracing with nanoscale molecular context. E11 bio releases some exciting results on brain mapping. Andy Mckenzie gives a longer explanation here.

Ultra-safe nuclear thermal rockets using lunar-derived fuel. Melt lunar regolith and the small amount of thorium dioxide in it will sink to the bottom. You can then breed this into uranium for fueling nuclear rockets in space. Avoids the problems of sending a rocket full of nuclear material from Earth.

1. And search? Surely you could adapt vector databases used in RAG to the sum of vectors that constitutes a low-rank matrix.

2. Can you tell I’m becoming LoRA-pilled?