p(doom) = 0 (thoughts on the imminent AI apocalypse)

We need to democratise security, to provide people and businesses access to the tools that protect them from the stupidity of that mindless cloud of data we call “Artificial Intelligence”

We are the priests

Of the Temples of Syrinx

Our great computers

Fill the hallowed halls

We are the priests

Of the Temples of Syrinx

All the gifts of life

Are held within these walls

We are apparently all doomed and there’s nothing you can do about it. In amongst the business-as-usual nonsense of politics and business a chap who nobody has heard from before posted on ex-twitter saying that AI was going to kill us all.

“A top safety researcher at Anthropic has warned AI is advancing so quickly he believes there is a greater than 10% chance it “could kill all humans” within the next decade.

Evan Hubinger said in a post on X, the risk from the models which currently exist was “low” but he was “worried” the technology might develop and improve itself soon to the point where it posed an existential risk to humanity.

He did not spell out how he thought AI systems could in future result in humans being wiped out.”

Now I don’t know a great deal about AI but there does seem to be a terrible tendency in our world to catastrophise just about everything. Mr Hubinger doesn’t explain just how the models become an existential risk to humanity but presents us with his p(doom) estimate of >10%. This isn’t an actual analysis of risk, of course, this is one individual’s guess about the risks to human survival from AI.

“These will soon be superhuman systems that can hack anything, revolutionise any field overnight, and acquire real power and resources.”

This sounds quite a lot more like a public relations exercise than doomster predictions of humanity’s demise. And the track record of millenarian cults is, to put it mildly, not a good one. The techbro alliance that’s promoting this p(doom) argument does so because it wants attention. Perhaps because real world applications of LLMs aren’t going quite as brilliantly as they predicted:

“...executives report little own-firm impact of AI over the last 3 years, with nine-in-ten reporting no impact on employment or productivity.”

So what do you do when your predictions of job-destroying, profit-boosting, efficiency-generating positives from spending money on AI turn out to be a tad undercooked? You double down. ‘The models are improving by the day, so much so that if you don’t give us billions we’ll hit the singularity and we’ll all die’. And they’re banking on us being frightened by arbitrary p(doom) guesses and that we don’t understand what’s going on under the bonnet of AI. I may not be alone in thinking that, just maybe, the emperor’s tailor is a charlatan.

But, to play along with the millenarians, let’s look at how the AI is going to kill us all. It seems there are two ways:

1. AI could make it easier for bad actors to create biological, chemical or other weapons

2. Rogue systems could attack infrastructure at unprecedented scale while evading human oversight

The first of these is, let’s be honest, not a technology issue but a humanity issue and sort of assumes that ‘bad actors’ want to destroy humanity. But outside film and comics the ‘bad actors’ don’t want to kill everyone, they want specific human outcomes related to power and resources. Moreover for AI to obtain these weapons it would require physical resources (10101010 only goes boom inside your computer, real world boom requires some chemistry or biology). This doesn’t eliminate the risks of ‘bad actors’ securing weapons of mass destruction but it does contain those risks: the ‘bad actor’ isn’t Thanos, he is motivated by prosecuting his and his group’s objectives. Which probably doesn’t involve killing off all of humanity.

Which gets us to the second way AI is going to kill us. This is more feasible since we remain inside the 10101010 world, infrastructure is increasingly dependent on computerised systems. But this doesn’t kill us, it just collapses the systems that sustain civilisation. Millions of people live without a reliable electricity supply, without piped water, and without access to the place where AI operates. Their lives aren’t great but, despite the problems, they aren’t dead yet.

There’s a lot of *intelligent* chatter about hierarchies of power and how these AI models will become ‘superintelligent’ but being able to add up quickly didn’t make computers sentient and being able to search through the vast online archive very quickly also doesn’t make AI Models sentient. Without human input (see ‘bad actors’ above) AI Models are just scratching their balls and staring out the window, they have no reason (obviously) to ‘attack’ infrastructure unless some human tells them to do so. What we should be concerned with isn’t having a boondoggle to agree an international treaty but with infrastructure owners having the capacity to use AI to stop ‘bad actors’ using AI to attack their critical systems.

It is impossible to dismiss the significance of AI and that it is going to make profound changes to your job and to the structure of business and industry. It is odd to see the emerging marriage between the big AI businesses and the political left as the former seek money to fund their pretended threat to humanity and the latter embrace a ‘neo-luddite’ agenda. In the middle we have people like Darren Jones, former cabinet minister, calling for meetings and treaties:

“I’ve never thought we should ban innovation or scientific endeavour but it’s clear we need a new multinational treaty for the safe and regulated development of superintelligence”

I’m not convinced by the p(doom) arguments. Most of these seem opportunistic and better placed in the realm of science fiction. But we do need to recognise that the AI genie is out of its bottle and that the structures of our civilisation are fragile. Without wanting to come over all Marxist, we need to democratise security, to provide people and businesses access to the tools that protect them from the stupidity of that mindless cloud of data we call “Artificial Intelligence”.

“To minimize suffering and to maximize security were natural and proper ends of society and Caesar. But then they became the only ends, somehow, and the only basis of law—a perversion. Inevitably, then, in seeking only them, we found only their opposites: maximum suffering and minimum security.”