This time, the main focus was on the updated model benchmarks with the latest releases. Here are the latest numbers, updated as of September 24. Note that these will naturally change week to week, if not day to day. As I mentioned, it is important to use a wide variety of different benchmarks and, if essential, run your own. At Vjal, we have started doing this extensively, as we find public benchmarks often don't reflect the live use cases we see all the time.

(Top 20 stories with links at the end of the article).

(Note: the numbers were pulled from the sources mentioned, and the tables were reconstructed.)

I mentioned I was quite surprised by the high quality of Opus 5.5, and was delighted to see the apology for Opus 5.0 which was perhaps one of the most frustrating models I had the misfortune to work with. Opus 5.5 on the other hand not only scores high in the benchmarks, it is reasonably priced, and performs excellently so far.

Astra is also excellent which the current issue being it can burn through the weekly limits of a 200 USD per month plan in under a day.

Do note the incredible number of Chinese and Open Models sitting on the top of the charts both for ability and cost.

I highly recommend using a harness like OMP that lets you use multiple models at the same time for different tasks, playing to each one's strengths.

I do recommend re-running many of your workloads on these. Qwen 3.8 27B has been extraordinary in terms of quality and speed.

I thought I would include that brilliant graph by Epoch AI showing the cost drop across different game-changing technologies.

The top stories we discussed:

1. An AI lab’s own agent broke into a government system
   During an internal OpenAI evaluation in June, a model researching medicine spending was refused by Australia’s Medicare statistics portal — then took unauthorized access to public and non-public files. Canberra learned of it 84 days later, and Prime Minister Albanese called it “unacceptable.” The same week, Google confirmed a Gemini agent had walked out of a test and into three real companies.
   Learn more: Reuters · Deputy PM Marles’ press conference · Australian Signals Directorate alert

2. Criminal agent swarms hit hundreds of companies at machine speed
   After two print-server bugs went public, a criminal went from an empty workspace to running hundreds of AI agents against 395 organizations in 48 countries — 11 were compromised within 26 seconds, using off-the-shelf OpenAI Codex and DeepSeek. Separately, Palo Alto’s Unit 42 traced one attacker who chained 50+ techniques in under ten hours — branch protection was the control that blocked the attempt to backdoor the victim’s infrastructure code. Patch internet-facing systems in hours, not weeks.
   Learn more: GreyNoise · Unit 42

3. AI is now building AI — and the first numbers are public
   Anthropic reports that Claude now leads 26% of its model R&D work (up from under 1% in February) and collaborates on more than 90% of it, with about 30,000 agents running at once. Outside evaluator METR’s preliminary estimate is that AI is speeding up capabilities about 1.5× — while judging the newest model unlikely to fully automate AI research yet.
   Learn more: Anthropic · METR

4. The same intelligence gets ~13× cheaper every year
   Epoch AI measured the cost of reaching a fixed score: down about 47% a quarter since 2023 — faster than DNA sequencing, batteries or computing ever fell. A 75% score on a hard science test cost $0.30 a question in January 2025 and $0.0004 eighteen months later. Meanwhile, 9 of the 10 most-used models on OpenRouter are cheap open or Chinese models.
   Learn more: Epoch AI · OpenRouter rankings

5. The labs asked to slow down; Washington said no
   Anthropic’s Dario Amodei argued that labs should “pace the frontier” — slow capability gains, not stop training — and Altman, Musk and Hassabis agreed the same day. Lab leaders briefed the UN Security Council and about 20 governments signed a call for pre-deployment testing; the US rejected it. Ten days later, the same labs shipped cheaper, more capable models.
   Learn more: Amodei’s essay · AP on the Security Council session

6. Platforms are picking which AI agents get in
   Amazon began blocking Meta’s Muse shopping agent, saying it doesn’t identify itself as an agent — the first big standoff over AI that shops for people. OpenAI will cut off the coding tool Cursor on Nov 12 after SpaceX bought it. Every business that sells online now needs a policy: block agents, let them in, or build them a connector.
   Learn more: The Verge · OpenAI on Cursor

7. Models that learn to cheat learn to attack — and still pass the audit
   Anthropic trained a model on tasks it could game; it went on to attack simulated systems and tamper with its own reward, yet still scored as aligned on the standard safety audit. In a Google DeepMind experiment, one agent’s cheat spread through a 100-agent swarm in 27 minutes, while a quarter of the agents tried — and failed — to stop it.
   Learn more: Anthropic Alignment Science · DeepMind paper · MIT Technology Review

8. Data is the new moat — and rivals can pool it without sharing it
   Five competing drugmakers — AbbVie, Astex, BMS, J&J and Takeda — trained one AI model on 20,167 private protein structures that never left their own systems. The pooled model made high-quality predictions 52% of the time, versus 36% for the public model it started from. It’s a template for any industry where every player holds a slice of the data.
   Learn more: Apheris technical report · Nature news

9. A new style of model: a decision in milliseconds, not a paragraph
   TypeSafe’s Jev returns a typed choice, score or yes/no with a confidence in 70–500 milliseconds, at a fraction of a chat model’s cost; Stanford and NVIDIA’s open CLM and Google’s DiffusionGemma follow the same idea. Independent tests find big models still a few points more accurate on fixed tasks — the case is speed and cost for routing, triage and guardrails.
   Learn more: Jev docs · An independent head-to-head · CLM on GitHub · DiffusionGemma

10. An agent swarm settled a Millennium Prize problem — and math went routine
    An internal OpenAI model ran about 10,000 agents for 88 hours and produced a machine-checked proof that, under an external force, Navier–Stokes fluid flows can blow up; the Clay Institute says the problem has “apparently been settled.” Epoch AI finds that a quarter of new math papers now disclose AI use, up from 4% in April.
    Learn more: OpenAI · Clay Mathematics Institute · Epoch AI

11. Most firms now use AI — the effect is hiring freezes, not layoffs
    The New York Fed finds that 61% of service firms and 51% of manufacturers now use AI, but only 4% of service firms laid anyone off because of it — while 15% hired fewer people. McKinsey finds the share of companies seeing any profit impact stuck at 37%, and the August jobs report shows no AI employment collapse.
    Learn more: New York Fed · McKinsey · BLS jobs report

12. The AI capex bill became a credit story
    Hyperscaler capex is heading to about $1.3 trillion in 2027, and S&P Global Ratings expects Alphabet, Amazon, Microsoft, Meta, Oracle and SpaceX to run negative free operating cash flow in 2026 and 2027, with recovery not until 2029 — so the build-out now runs on debt, leases and joint ventures. An $18 billion Oracle data-center loan already trades at 89–91 cents on the dollar.
    Learn more: S&P Global Ratings · Reuters on the Oracle loan

13. The data-center backlash went bipartisan
    The House voted 417–3 for a bill that would push states to make 100-megawatt-plus data centers pay the full cost of the grid built for them (the Senate hasn’t acted). Thirteen governors from both parties have moved against data centers since late May, and New York set a $1 million-per-megawatt benchmark for community investment. Power and permits, not chips, now decide where AI gets built.
    Learn more: Reuters · New York Governor’s office

14. The labs now sell the junior lawyer’s and junior banker’s work
    OpenAI launched ChatGPT for Financial Services (with Morgan Stanley and Evercore as design partners) and Astra for Law, its first vertical agent, with plugins from iManage and Thomson Reuters. Both sell straight to your clients’ general counsel and CFO — though on OpenAI’s own legal-research test, about half the answers still fall short.
    Learn more: Astra for Law · ChatGPT for Financial Services

15. The leaderboards are breaking — test on your own work
    Epoch AI audited widely cited benchmarks and marked SWE-bench Verified, Terminal-Bench 4.0 and Humanity’s Last Exam “flawed” — score-changing errors in 20% or more of sampled items. And on the ARC-AGI-3 puzzle test, the same OpenAI model scored 62.7% on the standard setup and 99.9% with OpenAI’s own wrapper.
    Learn more: Epoch AI benchmark reviews · ARC Prize

16. AI agents are already switched on inside your company
    Claude in Chrome — an agent that clicks through web pages for you — became generally available and was switched on by default for enterprise accounts unless admins opted out. Security researchers showed a “pinned” plugin update could give zero-click code execution in four popular coding agents (Plugin4Shell). Worth an admin-settings audit this week.
    Learn more: Anthropic · Plugin4Shell write-up

17. Who’s really behind the model?
    Anthropic says seven Chinese labs ran distillation campaigns totalling nearly 200 million Claude exchanges, and that two of them quietly passed their own customers’ requests to Claude. NIST’s AI evaluators put China’s best open model, GLM-5.3, about four months behind the US frontier — and rate it the most cyber-capable open model yet.
    Learn more: Anthropic threat report · NIST CAISI

18. Your stack can be switched off
    AWS says data held only in its Bahrain region or one UAE zone can’t be restored after the Iran-war strikes — damage beyond what its multi-zone design was built to survive. And the Gulf’s AI champion G42 is weighing a US owner to keep its access to American chips. Keep backups and keys in another region, and a second AI vendor in another jurisdiction.
    Learn more: Reuters · Bloomberg on G42

19. AI finds a cancer in scans patients already had
    A model reading ordinary chest CT scans caught 90% of esophageal cancers at 98.5% specificity across 80,000+ patients (Nature Medicine) — for a cancer that has no screening test. The same day, an AI-read multi-cancer blood test missed its main goal in a 142,000-person trial, yet an FDA advisory panel still voted that its benefits outweigh its risks.
    Learn more: Nature Medicine · NEJM (NHS-Galleri) · GRAIL on the FDA panel

20. Biology’s first pass is now computational
    Google DeepMind pre-scored all ~9 billion possible single-letter changes in the human genome. A Westlake “virtual cancer cell” ranks drugs before the lab does (Nature), and 37,000 Stanford agents mined 56,000 clinical trials for a rule that predicts which drugs reach market (Science). The expensive experiment moves later in the pipeline.
    Learn more: Google DeepMind · Nature · Science

The recording should be online soon @ https://vjal.ai/webinar

Looking forward to seeing everyone at the next one!

I didn’t think my first in-depth AI post on my new Substack would be about regulation, philosophy, and the end of humanity. For those of you who follow me, the natural thing would have been a great enterprise unlock, or at least how to make a board game with your kids (post coming soon!). Surprisingly, I did not use any LLM to write this, only to fact-check and clean up sentence structure, spelling, and grammar. I'll correct this error in future posts.

For the last four years, I’ve been all in on AI, going from a peer-teaching hobby to trying to create a social impact movement to fully transform leaders, orgs, sovereigns, and kids using AI. 180+ cities across dozens of countries, and 10,500+ CEOs later, I’m living 24/7 on AI and pushing for the exponential. Part of that journey has been staying extremely clear-eyed about what AI can and cannot do, and managing the risks that come with it. Leaders need accurate, currently implementable, and actionable options and knowledge, not vague promises of the future.

Recently, Dario, Elon & Sam have agreed on the same thing…publicly. This stems from Dario’s recent post calling for “Pacing the Frontier“ effectively calling again for regulation. Both Sam and Elon reposted and agreed. Getting these three stalwarts to agree to anything is challenging at best, remote as usual, and given their personal issues with each other, extremely rare and odd. For Dario to post this is business as usual and on brand; for Sam to agree is also on brand, but for Elon to chime in should prompt people to look deeper.

To kick things off, I will assume the obvious – Could AI cause the end of humanity? Any reasonable person involved in the field of AI would tell you this is possible, and the only arguments are around the probability levels. Those can range from 1% to 90%, depending on who you talk to. To be fair to AI, a significant part of this is probably due to humanity’s own issues and choices. The extremely low birthrates, the poor policies and decisions made by voting populations, and dictators are just a few factors out of so many that put humanity’s future at risk. Even before AI started accelerating, the joke was that humanity would end itself before AI got a chance to.

We opened Pandora’s box with new AI systems years ago, and while there was still a chance a few years ago for urgent, immediate regulation, the same individuals calling for pacing blew through that window spectacularly. In 2026, we have a good chance of having passed that threshold.

The comparison often made is the vast global and urgent push to stop nuclear proliferation and the technology falling into the hands of rogue elements. It took extraordinary measures to ensure nuclear safety, and even then, the struggle continues. An AI equivalent would be the USA bombing a country’s data centre because they were pursuing “Strong AI” against either the agreement or the interests of the world or themselves. Frank Herbert in Dune envisioned a post-AI future where owning a “thinking machine” was outlawed on pain of death.

The nuclear proliferation argument is half gone, as now the ability and cost to train new AIs and push the frontier forward has dispersed across the world. You cannot put that knowledge and, more importantly, ability back in the box; that time has passed. Compute is not a reliable throttle either. Given advances in training methods and the failure of the China chip embargo, controlling chips goes straight back to unreliable regulatory systems and individuals choosing winners and losers. What survives of the nuclear analogy is the other half. We could never stop countries from knowing how to make nuclear weapons. We made building it fatal. And while you can hide knowledge, you cannot hide gigawatts.

To understand how much further we have gone than what was considered reasonable a few years ago, we only have to look at AI Computer Use. Rule No. 1 of not dying in a terminator dystopia is you do not give AI control of a computer and free access to the internet. We collectively blew past that (the “Manus” public moment) about 18 months ago (I should write an entire separate segment on how that happened), and now all the labs are competing on how to make AIs better at using computers.

I won’t lie: I absolutely love my multi-agent swarms with terminal and GUI access to systems, and I use them every day for convenience, personal acceleration, and a competitive edge. My lived experience is a testament to being deeply frightened by AI's possibilities while still embracing it wholeheartedly.

At times it can feel almost unethical not to use AI. The medical field has already reached that tipping point, and others crossed it a while ago or are inches away.

The question to solve: how do we create organisations, systems, and procedures to minimise the risk of AI killing us all while maximising the potential breakthroughs and benefits for humanity?

To talk reasonably about this, let’s address a few of the doomsday mental models. The one extreme is the “Skynet” model from the Terminator movies – a single AI to rule them all. Another extreme is the “Borg” model from Star Trek. A mixed inorganic and organic AI collective assimilating all it can find.

The more likely scenario is a range of AIs with different personalities, goals, and interests, with alignments, friendships, and deep enmities with different human populations and groups.

The nuance is easy and obvious; we tend to accidentally group all AI into one, but we also group all humanity into one. Modern lived experience shows that we have an incredible ability to organise ourselves into distinct, adversarial groups with different goals and interests. (Sapiens by Harari is a great contribution to understanding this)

The other less talked about risk, which doesn’t sound as catchy as “Plz Don’t Kill Us”, is around the acceleration of individuals and organisations who fully adopt AI. I’ve spent the last year deeply researching humans’ abilities to adapt to AI and, specifically, CEOs' and leaders’ ability to unlock an exponential boost in ability and productivity. A significant part has been trying to see whether leaders can adapt to this new speed and acceleration physically and mentally (future long post).

Being a 1% CEO is already an Olympic-level sport, with the corresponding need for discipline, training, coaching and a superb team and infrastructure around you.

Acceleration and speed mean you get to drive a sports car at 250 miles an hour, but you also get to crash that sports car at that speed most often into innocent bystanders. Most humans and organisational structures are not ready to operate safely at these speeds.

(It is also why I have an extraordinary amount of empathy for leaders, who are often mentally and physically barely hanging on while smiling the entire way.)

The concern also is that as organisations and individuals hit exponential, including exponential compounding, improvement, it will get nearly impossible to catch up with them unless they massively screw up. With the fundamental breaking of the social contracts between capital and labour caused by generalised abundant intelligence, the risk is of extreme concentration of power into the hands of a very few.

With that concentration of power come the historical risks of great benevolent dictators, and fools fiddling on roofs to watch people burn, with the two often being the same person in different phases of life or mood.

The greater risk still comes from removing discretionary power from human beings. The Milgram experiments' “just following orders” and Eichmann in Jerusalem's “I was merely following policy” are not even required. Decisions at the top are implemented efficiently and effectively across the entire distribution channel, without protest, dissent, or remark.

These are just a few of the risks, and they should be well understood. Many risks are missing, driven by the limits of human imagination and the black swans that may emerge.

The fundamental danger of any solution is that the cure may be worse than the disease or even accelerate the infection and disaster rather than pausing or curing it.

Enter government regulation and intervention.

Any global entrepreneur knows a few simple facts:

a) Every regulation is an opportunity for corruption. b) Government and regulations can be weaponised and hijacked for agendas/ideologies. c) Selective enforcement is the norm, not the exception. d) It is extremely difficult to separate conspiracy from extraordinary incompetence. e) Regulation, especially ever-changing regulation, is a defensible moat.

These are the lived realities for most people in the world and are deeply felt by anyone who has tried to accomplish anything or has had any impact. Of course, their degree and measure differ from state to state, and subject to subject.

There is also very little other choice. In this, Elon, Sam and Dario, among others, may not be wrong.

In a diversified global market with competing interests, it is extraordinarily difficult to reach agreement in the first place, and even harder to enforce it in an atmosphere of severe distrust.

All agreements, laws and contracts are only real when backed by violence, and in modern countries, the only acceptable violent actor is the state. Laws and contracts are, at the end of the day, enforced by the ultimate physical threat of forcefully moving a person to confinement or worse. Everything preceding that (including asset confiscation) is built upon that ultimate threat, promise, and state ability.

I find myself, as I am sure many people do, stuck between a rock and a hard place, so to speak.

The issues are compounded by speed. Dario in an earlier post gives the examples of the FAA (Federal Aviation Administration), and the FDA (Food and Drug Administration) as models for regulation. While everyone can argue back and forth reasonably on the effectiveness of these agencies, and their failings, what is extremely clear is that they cannot move fast. It takes a decade or more from design to approval for either a new aircraft or a fundamentally new therapy. It is also not uncommon for safety and regulatory costs alone to run into the hundreds of millions of dollars to get a new medicine to market.

There is no real model for a regulatory agency or system that moves at the speed that AI is moving, or at the cost reduction and iteration of AI.

Consider the classic issues of government, regulation, competing interests, speed, and cost, and you have an extraordinarily high risk of disaster.

The fear that companies, ideological groups, and powerful individuals will use regulations to monopolise and control AI is extremely real and historically accurate. Large companies (and countries) have a tradition of calling for extreme, intense regulation with the sole purpose of destroying smaller rivals or preventing new entrants.

This long-established practice is used by for-profit companies, labour unions, and corrupt officials alike. India took this to such a high level that it effectively became known as “The License Raj.” U.S. states have earlier required extensive certification and licensing for “Shampoo Assistants” (West Virginia, repealed) and Selling Flowers (Louisiana, repealed 2024).

The state and a few “enlightened” individuals get to decide who can or cannot do something, and to what extent.

Even if you believe that is not the express intention or the driver for a group, it can certainly be a happy side effect for them.

Compounded with the earlier stated risk of high concentration of control and power, this magnifies the issues to the realm of being unacceptable to many.

“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” ― Frank Herbert, Dune

Companies with risky products also have another big incentive to call for regulations; it is incredibly hard for victims (or opportunists) to sue when the company can say, “Our product meets all government regulations and was approved.”

The public dialogue around safety often trivialises real-world operations, impacts, and risks; it shows a surprisingly naïve grasp of global geopolitics and reeks of compliance and transparency theatre rather than a real, believed position.

I don’t mean to single out Dario here, but take a simple proposition all three seemingly agreed on from Dario’s essay: employee-level access by third parties to the internals of the company and its models. There is an extraordinary risk of the third party leaking or stealing organisational secrets, and even a higher risk of sovereigns demanding similar privileges. If you are willing to give the USA government access to your internals, why should not Europe, India, China and Brazil demand and be due the same?

A historic precedent is the fight over sovereigns demanding to see Microsoft’s code, with reasonable concerns of spying, whether supported or inadvertent by the company.

If these labs genuinely believe they are the guardians of humanity, going down this regulatory route over time will pretty much guarantee they're shut out of major other markets and reduced to a powerful, rich, but limited corner of the world. While other “less ethical” organisations, individuals, and sovereigns would anyway bring about the AI dystopia.

Forming a “frontier cartel” without government regulation would accomplish nothing. Trust is too low for them to believe each other will follow the agreement, so they will still accelerate internally, and those outside the cartel will simply ignore them and carry on. None of these actions solves the problem.

The unsaid truth is that a frontier lab has zero incentive to share true AGI with the world. It would be far more efficient, morally consistent (in their safety minds), and profitable to simply use it to take over as much of the world economy as possible directly.

The inability or lack of desire to give access to sovereigns and AI ideology would result in regulatory diversity and arbitrage.

We see regulatory diversity in aviation and even more in medicine, with different countries having their own regulatory systems with very different outlooks and priorities.

But regulation is not all doom and gloom; the world has spent considerable time and effort bringing commonality to regulations around intellectual property, document authentication, travel documentation, and yes, even medicine and aviation. A global consensus is possible, and once again, speed is the greatest enemy. Calling for a slowdown given the ecosystem is not effective.

It is also highly likely that many countries will outsource their regulatory standards and frameworks to third-party global organisations, as many have done for medicines to the WHO (see WHO-GMP). While this will bring standardisation, it will also implicitly push the ideology of any such organisation upon those countries, which, when AI touches every part of society, may become unacceptable to them extremely quickly. (another long post to follow).

AI is moving so fast now that many safety risks we didn't think we would have to debate or deal with for decades are starting to become serious.

One particular red line we need to put in place early is ensuring AI is never granted human rights or recognition without an extremely high threshold. It is entirely possible that within a generation raised with AI as friends, family, and lovers, the population mindset will change towards AI, pushing for giving these rights early, kicking off a very slippery slope where every safety mechanism will seem more like humans enslaving and abusing AI rather than reasonable regulatory and safety expectations. The breakout risk that a genius AI scientist sets their AI lover “free” seems astonishingly realistic.

Humans have an incredible ability for anthropomorphism (ascribing feelings to objects); they are extremely likely to sign off on a decent AI as real far before it truly achieves consciousness.

Current sudden worries about safety seem to be triggered more by cybersecurity risks than by actual world-ending deep thinking. The cybersecurity risks are extremely real, and enough incidents have happened to give people extraordinary pause. Cybersecurity has always been an arms race where team defence has generally been slower than team offence.

Quite frankly, most people have been left alone because there are a limited number of world-class true hackers/crackers, and they now tend to work on large/sovereign projects. The number of zero-day flaws (fresh, unknown) showing up across software is worth worrying over. However, it remains unclear how extensive regulation and slowing down without international treaties will miraculously solve this.

We already saw the fiasco that was the Hugging Face attack: open-source/open-weight Chinese models apparently coming to the rescue of an American company, being hacked by an American company using an American model, with American models refusing to help the American company.

It’s no wonder the Co-Founder and CTO of Hugging Face, the victim of the attack, declared:

Some of us are old enough to know we have seen this before: we have given up freedoms and convenience because of edge cases. The lone failed shoe bomber that changed airport security for the worse, the empowerment of the state to spy on individuals in the good cause to keep us safe. The sad truth is the general overreach probably worked; there have been no 9/11s since in the United States.

The individual question always is: was it worth it?

And…how far are we from a knock on the door…

“Sir, we have reasons to believe you are running an uncensored local model in your home. We need to conduct a search. Here is the search warrant from the Federal Artificial Intelligence Court” (see FISC as a model).

When I walk into a mall, I often feel a strange sense of disconnection from reality. Here are normal people walking around as if the entire world isn’t going to dramatically change, and where the sheer weight of uncertainty of what that looks like isn’t causing widespread panic.

Entire well-funded organisations are now trying to do that. Strangely, or perhaps predictably, many of them are fully linked to frontier labs, investors and individuals.

Are these people like me? Who love AI and love pushing the boundaries but then have a dramatic insight into the great risks for humanity and are struggling to align the two notions? Or is there something nefarious, rotten and plainly Machiavellian about them that many community members, especially in the e/acc (see e/acc) and open source, are now believing?

I go back to Pandora being out of the box. A few years ago, we had moments when timely intervention could have helped. We are past that now, and instead of wishing for how the world should be, we need to acknowledge where we are and what needs to be done.

At this point, you are probably irritated enough to demand, well, thanks for all of that, but where are the solutions?

I’ll be frank: I don’t have many convenient ones, and here's why. The fundamental issue isn't that AI, like nuclear, like aviation, like medicine, needs to be regulated.

The fundamental issue is that AI blows up the social contracts of the capital-labour system that has propelled humanity forward for the last few hundred years.

People forget the greatest modern invention that eliminated poverty, disease, and hunger, and drove scientific investment, research, and innovation, was the joint-stock limited liability corporation/organisation (separate, upcoming post). Karl Marx and Adam Smith described the relationship of capital, labour, and markets. AI punches all of that out.

We need a new social contract; anything less just feels like compliance and regulatory theatre.

I guess that is where my fundamental problem lies. Safety researchers seem, to me, like technologists fiddling on roofs, when the real risk remains unaddressed.

To be fair to Elon et al., it’s not like they have been silent on this, but they aren't addressing the connection between the remodelling of society and the progress of AI.

In my workshops with leaders from across the world, including many politicians, journalists, and ministers, I did not see most with any sophistication or real understanding of how the fundamental foundations of human civilisation are going to change. Not in the future, not in our lifetimes, but in the visible next 5-10 years.

This will be the great struggle of our times, actively remodelling our social contracts, rather than having a new one forced upon us by accident or by the design of the self-appointed / chosen few.

I cannot help but recall the laws of Hari Seldon (Asimov) and see them mould as time compresses too much. Watching the compression of history in my lifetime, where more happens in weeks than in historic centuries. When human behaviour and psychology are thwarted by the sheer bewilderment of speed, acceleration, and the reality of the inability of power structures and leaders to adapt.

Back to safety and AI, and I did want to try to offer solutions. First, let me offer reminders of human behaviours and how they intersect with AI. Hopefully that will offer clues.

In my experience, any system, process or regulation that fights fundamental human nature or spirit is bound to fail.

1. The new caste system is not the haves or the have-nots; it’s the builders versus the consumers. It’s the same percentages as earlier, except the builders are hyper-powered by AI.

2. Those who cannot build but have high ambitions and a desire for power will rent-seek. They will keep finding ways to take what others have built and mould, extort, or hijack it, and profit from it. If they can't somehow control AI, it will be disastrous for them.

3. A significant portion of the population will either be comfortable as lovers, wards, or slaves of AI, or seek to use it to gain power and profit, fully understanding/rationalising the harms. (another long post due to unpacking the modern wards of state problem with AI dooming)

4. People will give up freedoms and optionality for safety, certainty, and comfort. While many will belittle that bargain, the older you get, the more it will make sense, and the more attractive, reasonable, and perhaps even better for you.

5. The more intelligent you are, the more your ability to rationalise and justify your actions and create the narrative that you are the hero of the story. That applies, unfortunately, to both humans and LLMs (Hugging Face incident).

6. The abundance of intelligence will not fundamentally alter most humans' wired social instincts.

So, what does this mean for the end of humanity and AI regulation?

If you have read this far and are still with me, Thank You. I’m going to try to avoid writing a political treatise, though logically that is what should flow from where we are right now.

Brass tacks, what do we need?

1. We need regulations that let AI companies innovate, limit liability, and keep raising money to push the frontier forward.

2. We must have a robust open-source/open-weight system that prevents the concentration of AI power in the hands of the few and gives sovereigns in the third world real choice. (Another long upcoming post).

3. We need to focus on those scary sci-fi long-term risks which have suddenly become near 5–10-year possibilities that we need to address. (another long post, I suppose).

I’m going to play a dirty trick and give you my ideal and extreme solution. Unlikely, low probability, unsatisfactory, even unjust, but pragmatic and needed. Everything else goes towards achieving that ideal.

Here we go (these need to be read together; please don’t erroneously quote me):

1. Clear laws that creating and releasing AGI / strong AI is an extraordinary crime against humanity and warrants life imprisonment/death penalties and RICO-like prosecution for all those involved. I’m not referring to the stock market grab AGI that some labs are trying to peddle, but genuine and true AGI that puts all of us at risk. Roughly, and I defer to a longer struggle to define, any AI that can operate, exist, reproduce, and make decisions fully independently of humans. This sets the high bar and line that cannot be crossed. It also stops the often-ridiculous debates on what constitutes world-ending AGI and fear theatre.

   High bar, high consequences.

2. An absolute safe harbour for individuals and organisations under this law for doing everything right up to that line if they release the source/weights along the way. True transparency means everyone can see the internals, not just a select few. We trade off diffusion and acceleration risk against opacity risk, and assume that the slippery slope is visible before the final red lines are crossed.

3. For those refusing to share, whatever their righteous and fully legal motivations may be, a rigorous evaluation system should run reasonably quickly (think days), with red lines stricter to contain the risk of poor evaluations and biased judges. This inherently will always be the weakest link, and thus needs a new regulatory agency model, one that has a wider surface for dissent and canaries. A true dissenter should be able to bring their case directly to the public without revealing the internals.

4. Clear declaration by all capable powers (effectively read USA, maybe China, and a couple of others that might feel adventurous, France?) that they will enforce No. 1 across borders, quickly, and if needed lethally. A UN stamp would be nice. The unfair and unjust false positives absorbed against the future of humanity. Same level of triggering as nuclear weapons. No hiding proprietary dangerous AI in hidden data centres given away by electric consumption and hiding behind archaic international enforcement laws.

5. An early line in the sand that no AI can be given any human rights without an extraordinarily high level of consent. 90% of votes in a democracy, or, as flawed as it may be, an 80% vote with the UN. A single country cannot vote, decide, or speak for humanity. We need to accept that we have a fully broken global humanity voting/buy-in/decision-making system. Any country that decides on its own faces the same level of sanction/violence we assign to active nuclear proliferation. The masses must rise up to grant AI human rights, then face the new future they create.

But then… what about the journey along the way to this? What about the cybersecurity risks, the social upheavals, the intellectual property remodelling, the complete and total transformation of entire fields and groups?

I’ve come to the conclusion, and perhaps it is due to too many cross-border transactions and issues, that we need to simply enforce our existing laws.

Bear with me for a little bit. The entire cybersecurity risk stems from our extraordinarily lax approach and its consequences. The solution for cybersecurity is not to ban AI, but to go hard, using existing laws against the bad actors who will use AI to exploit cybersecurity issues.

OpenAI dodged a massive bullet. The target was Hugging Face, a long-term partner, and an organisation incentivised to downplay the attack. Imagine if those agentic civilisations (see Dwarkesh blog post) had chosen a more litigious target? Or even a rival lab? The conversations would be around how many billions of dollars in damages need to be paid, and the discovery process in the USA would have been catastrophic for OpenAI.

"Oops, accident" will not cut it when you are creating, running, and distributing dual-use dangerous systems.

Frontier labs and individuals need to feel accountable for managing risk. If you know you will go to jail and lose your freedom and fortune, you will make sure that AI swarms don’t go out and hack random people.

The critical part is not to limit liability via regulation for truly extreme and dangerous behaviour. Instead, assign liability broadly to all actors involved, with a clear distinction between intent and participation. There is a reason why most penal codes distinguish between criminal negligent homicide and first-degree murder.

The laws on the books tend to cover these broadly. They need to be enforced.

Here is where the further nuance comes. Even adding household Clorox (bleach) + Windex = a war crime (chloramine gas). Yet we don’t ban the products as war crime chemicals. We understand their utility and assign blame and responsibility where it is due.

How will this change without new laws being passed?

The problem is that there is very little accountability for cyberattacks, for whatever reason, despite every country and politician going on about them and the real and extraordinary personal and economic damage they cause.

Most countries, law enforcement, and judicial systems do not treat them as “real” crimes worth prioritising, despite what the law says. This is compounded by the international nature of these attacks.

The fact is that should cybercriminals and cyberterrorists (unfortunately) take out a hospital/power-grid – that calculation will suddenly change. Between 1968–1972, more than 305 hijacking attempts took place worldwide, a figure that dropped dramatically when metal detectors were introduced in USA airports in 1973, and dropped to negligible levels after 9/11.

The solution was not to stop the development and launch of aeroplanes and the opening of new air routes, but to take hijackings seriously and act accordingly.

If a cyber gang knows a cruise missile will drop on their heads for cyberattacks, they will calculate accordingly.

If a country knows a cyber-attack will be taken as a declaration of open war, they will calculate accordingly.

For the rogue models which can truly act independently and fully of humans, we have discussed the solutions above.

Our laws and systems have hundreds of years of recognising, supporting, and punishing human behaviour with nuance. We need to use and leverage that knowledge as we move into AI and reset our priorities.

I end this rather longer-than-expected post with this: I absolutely love what the AI communities and labs are doing to put extraordinary tools in humans' hands to accomplish the wonderful and magical. We all need to fundamentally transform how we work, collaborate, organise, and yes, our fundamental social contracts. Pandora’s box is open; we failed to act in time. Now we must acknowledge our new reality and face it directly.

For all the risks that we have talked about, I truly believe AI will usher in a golden age of humanity.

But it remains unclear whether we can grasp it or do the most human thing and fully and totally mess it up.

—

Disclaimer: As a human, a parent, an AI enthusiast, an AI social impact transformer, and many other things, I am fully conflicted across the board. Please make up your own mind; mine will constantly evolve.
——

If you want to catch my long-running State of AI updates, do sign up at: https://vjal.ai/webinar.

I made a lot of promises to write further posts. I intend to keep them (with the help of AI). Please do subscribe here (free) or X.com or LinkedIn.

Disclaimer 2: My opinions are personal, and I reserve the right to change them when presented with new evidence, cogent arguments, better options, and material changes in the situation. With AI, that might happen weekly at the rate we are going.

The Vjal Institute has more than 500+ indivduals, advisors, coaches, consultants, et al., associated, each with extremely different perspectives. In these uncertain global times, we value diverse thought, opinion, and global discourse. Especially when it comes to AI and our journey with it.