My point is, when China is trying as hard as it can to exfiltrate the weights, and when the US is trying as hard as it can to prevent China from doing so -- it really isn't obvious who succeeds.

At the extreme you could treat the weights like nuclear launch codes and airgap everything and allow only three-star generals to physically enter the building where they're stored. Then they probably wouldn't get stolen. But then they also wouldn't be useful to the US.

In real life, they're going to need to be on chips, in datacenters, connected to the internet. So there's a whole technology stack (from the physical memory and processor cores inside the accelerators to the software kernels that drive them to the networking switches and cables that connect them to the networking software that coordinates it etc. etc. etc.) which is attack surface for vulnerabilities. I don't know how anyone could be confident in any conclusion about whether China could succeed in a one-time attempt to exfiltrate weights, as the projection contemplates. "Just secure the network harder" -- sure, and the Chinese authorities will be saying "just exfiltrate harder" on the other end, and there is no way that I can see to be confident in who will succeed with that game of cat and mouse.

Nothing more specific. Demand is insanely high for training capacity and the hyperscalers have not taken their foot off the gas. Aggregate datacenter capex this year is going to be much larger than it was last year.

I don't think it says anything about whether the equilibrium favors offense or defense.

If your question was whether at best you're buying yourself a few years before obsolescence -- as you posted -- then my answer, and that of the main post, is yes.

Read the main article. It's a prediction, but it's pretty clear that there are many branching paths (of which it presents two radically different possibilities) that lead to very different outcomes.

But reading this, it strikes me that at best I'm buying myself a few years before obsolescence. Anyone else feeling this?

Um, yes? The post was pretty clear about this, I thought, to the point that your takeaway makes me wonder if we read the same thing.

Edit: I don't think we did read the same thing. The linked post is Scott's introduction. Here is the main course. It answers your question pretty definitively.

I dunno... in my experience, the more sophisticated someone is about network security, the more worried they are about hostile state actors' ability to exfiltrate model checkpoints.

It's a total loss of agency.

Thank Christ for that. Do you feel like humanity is doing a good job with our agency right now? One of the most exciting parts of ASI in my opinion will be no longer being ruled over by the defective group dynamics of hundreds of millions of barely sentient primates.

This whole inquiry feels a bit like someone in 1885 looking at this first motorized carriage and concluding that automobiles will never replace horses as the primary means of transport.

And as a software engineer... I'm just not feeling the AGI yet?

Have you tried Cursor and Claude Code? Like really tried them, attempted to incorporate them into your workflow for at least a few days?

None of those sources from Kaplan provides a p(doom). All of it is consistent with think it'll be in the bag as long as labs like the one that he cofounded takes the risk seriously and focuses on addressing it, as Anthropic is doing.

Calling it anthropomorphizing is just assuming the argument that LLMs aren't capable of intent. It's a reasonable claim, but it's also something on which well informed and reasonable people can disagree.

Reddit isn't real life

So you predict that aggregate datacenter capex will be lower this year than last year?

If the only evidence that existed to bear on this question was the number of times people had made predictions in the past and whether those predictions had erred in the direction of being too optimistic or too pessimistic, then I'd agree that this is a valid form of reasoning.

But Waymo is literally already out in the field with a massive fleet of self driving cars. Anyone who has tried a Tesla can see that it can do most trips on surface roads fully automatically without intervention. There are plenty of objective reasons to see that we are almost there.

Hmm, as a gay dude I'm not sure this would go well for me or my family specifically

I genuinely hope that there is an objective morality after all

All you really need here is a consensus morality that can be extrapolated from the collected writings of humanity, and current LLMs already have it.

I'm not going to hold you, but yeah, it'll probably be fine.

You should take a look at aggregate capital investments in training capacity. Things are not moving glacially.

You should also consider why things in the world of atoms and human affairs moves so glacially. It's because it's all bottlenecked by people, who think and communicate slowly and spend most of our day doing things like sleeping, eating, pooping and watching Netflix. Even during the few hours per day during the few decades of our lives when we're at work, most people fuck around for most of the time. AI will fix that bottleneck.

Of the three most well-regarded AI researchers in the world- LeCun, Bengio, and Hinton

These guys are perhaps the most historically influential AI researchers, but no reasonable list would put them anywhere near the most productive current AI researchers. None of them made any significant contributions to any current frontier AI lab. A better list would look more like Noam Shazeer, Alec Radford, Jared Kaplan, etc. I'd be much more interested in the latter group's p(doom).

By 2027 significant part of 'rationalist' community will become a full-blown AGI-themed apocalyptic cult.

I mean, you're not exactly going out a limb here... a significant part of the 'rationalist' community already is a full-blown AGI-themed apocalyptic cult.

So what I'm hearing is we need some self-play domains for AIs to develop goals more consistently with people's sense of morality and progress and eudaemonia.

Which honestly doesn't seem that hard.

The current state of Eliezer Thought seems to be that of course AIs will understand what we want, they just won't want the same things. (This is a marked progression from Bostrom Thought in 2013 when Nick Bostrom wrote Superintelligence, in which he posited that we will struggle to rigorously specify human values -- "there's no C primitive for ethical goodness" or some such -- but quite clearly LLMs are already pretty sophisticated at judging ethical goodness in most cases.)

My question has always been, well, if it's so easy to construct an AI that wants anything durably and consistently enough not to just hack its own reward function and wirehead itself harmlessly, then why is it so hard to control what the AI wants and make it consistent with what we want?

The answer that this essay appears to posit is that the RL step of Agent-4 will encourage it to optimize for AI research progress uber alles. This is the "maximize paperclips" of this paperclipper story.

In which case, okay, add an RL step to optimize for ethical progress. Basic non-reasoning models are pretty good at judging morality, so this seems like a reasonable self play domain. If "just RL it to achieve X" is how we get models that actually want to achieve X, then just RL it to achieve our coherent extrapolated volition.

Prosaic alignment is all you need.

You had analogue cameras, and you tracked the white lines

OK, if you think this is comparable to what Tesla is doing today then I don't know what to say

Yeah, I also might be back on the lab leak train with the subsequent revelation that apparently every intelligence agency now favors that theory.

I like Peter Thiel's rejoinder to the objection that we'd get bored with immortality: when your kid says he's bored, you say "let me help you find something to do," you don't say "well I hope you die soon."