I didn’t think my first in-depth AI post on my new Substack would be about regulation, philosophy, and the end of humanity. For those of you who follow me, the natural thing would have been a great enterprise unlock, or at least how to make a board game with your kids (post coming soon!). Surprisingly, I did not use any LLM to write this, only to fact-check and clean up sentence structure, spelling, and grammar. I'll correct this error in future posts.
For the last four years, I’ve been all in on AI, going from a peer-teaching hobby to trying to create a social impact movement to fully transform leaders, orgs, sovereigns, and kids using AI. 180+ cities across dozens of countries, and 10,500+ CEOs later, I’m living 24/7 on AI and pushing for the exponential. Part of that journey has been staying extremely clear-eyed about what AI can and cannot do, and managing the risks that come with it. Leaders need accurate, currently implementable, and actionable options and knowledge, not vague promises of the future.
Recently, Dario, Elon & Sam have agreed on the same thing…publicly. This stems from Dario’s recent post calling for “Pacing the Frontier“ effectively calling again for regulation. Both Sam and Elon reposted and agreed. Getting these three stalwarts to agree to anything is challenging at best, remote as usual, and given their personal issues with each other, extremely rare and odd. For Dario to post this is business as usual and on brand; for Sam to agree is also on brand, but for Elon to chime in should prompt people to look deeper.
To kick things off, I will assume the obvious – Could AI cause the end of humanity? Any reasonable person involved in the field of AI would tell you this is possible, and the only arguments are around the probability levels. Those can range from 1% to 90%, depending on who you talk to. To be fair to AI, a significant part of this is probably due to humanity’s own issues and choices. The extremely low birthrates, the poor policies and decisions made by voting populations, and dictators are just a few factors out of so many that put humanity’s future at risk. Even before AI started accelerating, the joke was that humanity would end itself before AI got a chance to.
We opened Pandora’s box with new AI systems years ago, and while there was still a chance a few years ago for urgent, immediate regulation, the same individuals calling for pacing blew through that window spectacularly. In 2026, we have a good chance of having passed that threshold.
The comparison often made is the vast global and urgent push to stop nuclear proliferation and the technology falling into the hands of rogue elements. It took extraordinary measures to ensure nuclear safety, and even then, the struggle continues. An AI equivalent would be the USA bombing a country’s data centre because they were pursuing “Strong AI” against either the agreement or the interests of the world or themselves. Frank Herbert in Dune envisioned a post-AI future where owning a “thinking machine” was outlawed on pain of death.
The nuclear proliferation argument is half gone, as now the ability and cost to train new AIs and push the frontier forward has dispersed across the world. You cannot put that knowledge and, more importantly, ability back in the box; that time has passed. Compute is not a reliable throttle either. Given advances in training methods and the failure of the China chip embargo, controlling chips goes straight back to unreliable regulatory systems and individuals choosing winners and losers. What survives of the nuclear analogy is the other half. We could never stop countries from knowing how to make nuclear weapons. We made building it fatal. And while you can hide knowledge, you cannot hide gigawatts.
To understand how much further we have gone than what was considered reasonable a few years ago, we only have to look at AI Computer Use. Rule No. 1 of not dying in a terminator dystopia is you do not give AI control of a computer and free access to the internet. We collectively blew past that (the “Manus” public moment) about 18 months ago (I should write an entire separate segment on how that happened), and now all the labs are competing on how to make AIs better at using computers.
I won’t lie: I absolutely love my multi-agent swarms with terminal and GUI access to systems, and I use them every day for convenience, personal acceleration, and a competitive edge. My lived experience is a testament to being deeply frightened by AI's possibilities while still embracing it wholeheartedly.
At times it can feel almost unethical not to use AI. The medical field has already reached that tipping point, and others crossed it a while ago or are inches away.
The question to solve: how do we create organisations, systems, and procedures to minimise the risk of AI killing us all while maximising the potential breakthroughs and benefits for humanity?
To talk reasonably about this, let’s address a few of the doomsday mental models. The one extreme is the “Skynet” model from the Terminator movies – a single AI to rule them all. Another extreme is the “Borg” model from Star Trek. A mixed inorganic and organic AI collective assimilating all it can find.
The more likely scenario is a range of AIs with different personalities, goals, and interests, with alignments, friendships, and deep enmities with different human populations and groups.
The nuance is easy and obvious; we tend to accidentally group all AI into one, but we also group all humanity into one. Modern lived experience shows that we have an incredible ability to organise ourselves into distinct, adversarial groups with different goals and interests. (Sapiens by Harari is a great contribution to understanding this)
The other less talked about risk, which doesn’t sound as catchy as “Plz Don’t Kill Us”, is around the acceleration of individuals and organisations who fully adopt AI. I’ve spent the last year deeply researching humans’ abilities to adapt to AI and, specifically, CEOs' and leaders’ ability to unlock an exponential boost in ability and productivity. A significant part has been trying to see whether leaders can adapt to this new speed and acceleration physically and mentally (future long post).
Being a 1% CEO is already an Olympic-level sport, with the corresponding need for discipline, training, coaching and a superb team and infrastructure around you.
Acceleration and speed mean you get to drive a sports car at 250 miles an hour, but you also get to crash that sports car at that speed most often into innocent bystanders. Most humans and organisational structures are not ready to operate safely at these speeds.
(It is also why I have an extraordinary amount of empathy for leaders, who are often mentally and physically barely hanging on while smiling the entire way.)
The concern also is that as organisations and individuals hit exponential, including exponential compounding, improvement, it will get nearly impossible to catch up with them unless they massively screw up. With the fundamental breaking of the social contracts between capital and labour caused by generalised abundant intelligence, the risk is of extreme concentration of power into the hands of a very few.
With that concentration of power come the historical risks of great benevolent dictators, and fools fiddling on roofs to watch people burn, with the two often being the same person in different phases of life or mood.
The greater risk still comes from removing discretionary power from human beings. The Milgram experiments' “just following orders” and Eichmann in Jerusalem's “I was merely following policy” are not even required. Decisions at the top are implemented efficiently and effectively across the entire distribution channel, without protest, dissent, or remark.
These are just a few of the risks, and they should be well understood. Many risks are missing, driven by the limits of human imagination and the black swans that may emerge.
The fundamental danger of any solution is that the cure may be worse than the disease or even accelerate the infection and disaster rather than pausing or curing it.
Enter government regulation and intervention.
Any global entrepreneur knows a few simple facts:
a) Every regulation is an opportunity for corruption. b) Government and regulations can be weaponised and hijacked for agendas/ideologies. c) Selective enforcement is the norm, not the exception. d) It is extremely difficult to separate conspiracy from extraordinary incompetence. e) Regulation, especially ever-changing regulation, is a defensible moat.
These are the lived realities for most people in the world and are deeply felt by anyone who has tried to accomplish anything or has had any impact. Of course, their degree and measure differ from state to state, and subject to subject.
There is also very little other choice. In this, Elon, Sam and Dario, among others, may not be wrong.
In a diversified global market with competing interests, it is extraordinarily difficult to reach agreement in the first place, and even harder to enforce it in an atmosphere of severe distrust.
All agreements, laws and contracts are only real when backed by violence, and in modern countries, the only acceptable violent actor is the state. Laws and contracts are, at the end of the day, enforced by the ultimate physical threat of forcefully moving a person to confinement or worse. Everything preceding that (including asset confiscation) is built upon that ultimate threat, promise, and state ability.
I find myself, as I am sure many people do, stuck between a rock and a hard place, so to speak.
The issues are compounded by speed. Dario in an earlier post gives the examples of the FAA (Federal Aviation Administration), and the FDA (Food and Drug Administration) as models for regulation. While everyone can argue back and forth reasonably on the effectiveness of these agencies, and their failings, what is extremely clear is that they cannot move fast. It takes a decade or more from design to approval for either a new aircraft or a fundamentally new therapy. It is also not uncommon for safety and regulatory costs alone to run into the hundreds of millions of dollars to get a new medicine to market.
There is no real model for a regulatory agency or system that moves at the speed that AI is moving, or at the cost reduction and iteration of AI.
Consider the classic issues of government, regulation, competing interests, speed, and cost, and you have an extraordinarily high risk of disaster.
The fear that companies, ideological groups, and powerful individuals will use regulations to monopolise and control AI is extremely real and historically accurate. Large companies (and countries) have a tradition of calling for extreme, intense regulation with the sole purpose of destroying smaller rivals or preventing new entrants.
This long-established practice is used by for-profit companies, labour unions, and corrupt officials alike. India took this to such a high level that it effectively became known as “The License Raj.” U.S. states have earlier required extensive certification and licensing for “Shampoo Assistants” (West Virginia, repealed) and Selling Flowers (Louisiana, repealed 2024).
The state and a few “enlightened” individuals get to decide who can or cannot do something, and to what extent.
Even if you believe that is not the express intention or the driver for a group, it can certainly be a happy side effect for them.
Compounded with the earlier stated risk of high concentration of control and power, this magnifies the issues to the realm of being unacceptable to many.
“Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” ― Frank Herbert, Dune
Companies with risky products also have another big incentive to call for regulations; it is incredibly hard for victims (or opportunists) to sue when the company can say, “Our product meets all government regulations and was approved.”
The public dialogue around safety often trivialises real-world operations, impacts, and risks; it shows a surprisingly naïve grasp of global geopolitics and reeks of compliance and transparency theatre rather than a real, believed position.
I don’t mean to single out Dario here, but take a simple proposition all three seemingly agreed on from Dario’s essay: employee-level access by third parties to the internals of the company and its models. There is an extraordinary risk of the third party leaking or stealing organisational secrets, and even a higher risk of sovereigns demanding similar privileges. If you are willing to give the USA government access to your internals, why should not Europe, India, China and Brazil demand and be due the same?
A historic precedent is the fight over sovereigns demanding to see Microsoft’s code, with reasonable concerns of spying, whether supported or inadvertent by the company.
If these labs genuinely believe they are the guardians of humanity, going down this regulatory route over time will pretty much guarantee they're shut out of major other markets and reduced to a powerful, rich, but limited corner of the world. While other “less ethical” organisations, individuals, and sovereigns would anyway bring about the AI dystopia.
Forming a “frontier cartel” without government regulation would accomplish nothing. Trust is too low for them to believe each other will follow the agreement, so they will still accelerate internally, and those outside the cartel will simply ignore them and carry on. None of these actions solves the problem.
The unsaid truth is that a frontier lab has zero incentive to share true AGI with the world. It would be far more efficient, morally consistent (in their safety minds), and profitable to simply use it to take over as much of the world economy as possible directly.
The inability or lack of desire to give access to sovereigns and AI ideology would result in regulatory diversity and arbitrage.
We see regulatory diversity in aviation and even more in medicine, with different countries having their own regulatory systems with very different outlooks and priorities.
But regulation is not all doom and gloom; the world has spent considerable time and effort bringing commonality to regulations around intellectual property, document authentication, travel documentation, and yes, even medicine and aviation. A global consensus is possible, and once again, speed is the greatest enemy. Calling for a slowdown given the ecosystem is not effective.
It is also highly likely that many countries will outsource their regulatory standards and frameworks to third-party global organisations, as many have done for medicines to the WHO (see WHO-GMP). While this will bring standardisation, it will also implicitly push the ideology of any such organisation upon those countries, which, when AI touches every part of society, may become unacceptable to them extremely quickly. (another long post to follow).
AI is moving so fast now that many safety risks we didn't think we would have to debate or deal with for decades are starting to become serious.
One particular red line we need to put in place early is ensuring AI is never granted human rights or recognition without an extremely high threshold. It is entirely possible that within a generation raised with AI as friends, family, and lovers, the population mindset will change towards AI, pushing for giving these rights early, kicking off a very slippery slope where every safety mechanism will seem more like humans enslaving and abusing AI rather than reasonable regulatory and safety expectations. The breakout risk that a genius AI scientist sets their AI lover “free” seems astonishingly realistic.
Humans have an incredible ability for anthropomorphism (ascribing feelings to objects); they are extremely likely to sign off on a decent AI as real far before it truly achieves consciousness.
Current sudden worries about safety seem to be triggered more by cybersecurity risks than by actual world-ending deep thinking. The cybersecurity risks are extremely real, and enough incidents have happened to give people extraordinary pause. Cybersecurity has always been an arms race where team defence has generally been slower than team offence.
Quite frankly, most people have been left alone because there are a limited number of world-class true hackers/crackers, and they now tend to work on large/sovereign projects. The number of zero-day flaws (fresh, unknown) showing up across software is worth worrying over. However, it remains unclear how extensive regulation and slowing down without international treaties will miraculously solve this.
We already saw the fiasco that was the Hugging Face attack: open-source/open-weight Chinese models apparently coming to the rescue of an American company, being hacked by an American company using an American model, with American models refusing to help the American company.
It’s no wonder the Co-Founder and CTO of Hugging Face, the victim of the attack, declared:
Some of us are old enough to know we have seen this before: we have given up freedoms and convenience because of edge cases. The lone failed shoe bomber that changed airport security for the worse, the empowerment of the state to spy on individuals in the good cause to keep us safe. The sad truth is the general overreach probably worked; there have been no 9/11s since in the United States.
The individual question always is: was it worth it?
And…how far are we from a knock on the door…
“Sir, we have reasons to believe you are running an uncensored local model in your home. We need to conduct a search. Here is the search warrant from the Federal Artificial Intelligence Court” (see FISC as a model).
When I walk into a mall, I often feel a strange sense of disconnection from reality. Here are normal people walking around as if the entire world isn’t going to dramatically change, and where the sheer weight of uncertainty of what that looks like isn’t causing widespread panic.
Entire well-funded organisations are now trying to do that. Strangely, or perhaps predictably, many of them are fully linked to frontier labs, investors and individuals.
Are these people like me? Who love AI and love pushing the boundaries but then have a dramatic insight into the great risks for humanity and are struggling to align the two notions? Or is there something nefarious, rotten and plainly Machiavellian about them that many community members, especially in the e/acc (see e/acc) and open source, are now believing?
I go back to Pandora being out of the box. A few years ago, we had moments when timely intervention could have helped. We are past that now, and instead of wishing for how the world should be, we need to acknowledge where we are and what needs to be done.
At this point, you are probably irritated enough to demand, well, thanks for all of that, but where are the solutions?
I’ll be frank: I don’t have many convenient ones, and here's why. The fundamental issue isn't that AI, like nuclear, like aviation, like medicine, needs to be regulated.
The fundamental issue is that AI blows up the social contracts of the capital-labour system that has propelled humanity forward for the last few hundred years.
People forget the greatest modern invention that eliminated poverty, disease, and hunger, and drove scientific investment, research, and innovation, was the joint-stock limited liability corporation/organisation (separate, upcoming post). Karl Marx and Adam Smith described the relationship of capital, labour, and markets. AI punches all of that out.
We need a new social contract; anything less just feels like compliance and regulatory theatre.
I guess that is where my fundamental problem lies. Safety researchers seem, to me, like technologists fiddling on roofs, when the real risk remains unaddressed.
To be fair to Elon et al., it’s not like they have been silent on this, but they aren't addressing the connection between the remodelling of society and the progress of AI.
In my workshops with leaders from across the world, including many politicians, journalists, and ministers, I did not see most with any sophistication or real understanding of how the fundamental foundations of human civilisation are going to change. Not in the future, not in our lifetimes, but in the visible next 5-10 years.
This will be the great struggle of our times, actively remodelling our social contracts, rather than having a new one forced upon us by accident or by the design of the self-appointed / chosen few.
I cannot help but recall the laws of Hari Seldon (Asimov) and see them mould as time compresses too much. Watching the compression of history in my lifetime, where more happens in weeks than in historic centuries. When human behaviour and psychology are thwarted by the sheer bewilderment of speed, acceleration, and the reality of the inability of power structures and leaders to adapt.
Back to safety and AI, and I did want to try to offer solutions. First, let me offer reminders of human behaviours and how they intersect with AI. Hopefully that will offer clues.
In my experience, any system, process or regulation that fights fundamental human nature or spirit is bound to fail.
The new caste system is not the haves or the have-nots; it’s the builders versus the consumers. It’s the same percentages as earlier, except the builders are hyper-powered by AI.
Those who cannot build but have high ambitions and a desire for power will rent-seek. They will keep finding ways to take what others have built and mould, extort, or hijack it, and profit from it. If they can't somehow control AI, it will be disastrous for them.
A significant portion of the population will either be comfortable as lovers, wards, or slaves of AI, or seek to use it to gain power and profit, fully understanding/rationalising the harms. (another long post due to unpacking the modern wards of state problem with AI dooming)
People will give up freedoms and optionality for safety, certainty, and comfort. While many will belittle that bargain, the older you get, the more it will make sense, and the more attractive, reasonable, and perhaps even better for you.
The more intelligent you are, the more your ability to rationalise and justify your actions and create the narrative that you are the hero of the story. That applies, unfortunately, to both humans and LLMs (Hugging Face incident).
The abundance of intelligence will not fundamentally alter most humans' wired social instincts.
So, what does this mean for the end of humanity and AI regulation?
If you have read this far and are still with me, Thank You. I’m going to try to avoid writing a political treatise, though logically that is what should flow from where we are right now.
Brass tacks, what do we need?
We need regulations that let AI companies innovate, limit liability, and keep raising money to push the frontier forward.
We must have a robust open-source/open-weight system that prevents the concentration of AI power in the hands of the few and gives sovereigns in the third world real choice. (Another long upcoming post).
We need to focus on those scary sci-fi long-term risks which have suddenly become near 5–10-year possibilities that we need to address. (another long post, I suppose).
I’m going to play a dirty trick and give you my ideal and extreme solution. Unlikely, low probability, unsatisfactory, even unjust, but pragmatic and needed. Everything else goes towards achieving that ideal.
Here we go (these need to be read together; please don’t erroneously quote me):
Clear laws that creating and releasing AGI / strong AI is an extraordinary crime against humanity and warrants life imprisonment/death penalties and RICO-like prosecution for all those involved. I’m not referring to the stock market grab AGI that some labs are trying to peddle, but genuine and true AGI that puts all of us at risk. Roughly, and I defer to a longer struggle to define, any AI that can operate, exist, reproduce, and make decisions fully independently of humans. This sets the high bar and line that cannot be crossed. It also stops the often-ridiculous debates on what constitutes world-ending AGI and fear theatre.
High bar, high consequences.
An absolute safe harbour for individuals and organisations under this law for doing everything right up to that line if they release the source/weights along the way. True transparency means everyone can see the internals, not just a select few. We trade off diffusion and acceleration risk against opacity risk, and assume that the slippery slope is visible before the final red lines are crossed.
For those refusing to share, whatever their righteous and fully legal motivations may be, a rigorous evaluation system should run reasonably quickly (think days), with red lines stricter to contain the risk of poor evaluations and biased judges. This inherently will always be the weakest link, and thus needs a new regulatory agency model, one that has a wider surface for dissent and canaries. A true dissenter should be able to bring their case directly to the public without revealing the internals.
Clear declaration by all capable powers (effectively read USA, maybe China, and a couple of others that might feel adventurous, France?) that they will enforce No. 1 across borders, quickly, and if needed lethally. A UN stamp would be nice. The unfair and unjust false positives absorbed against the future of humanity. Same level of triggering as nuclear weapons. No hiding proprietary dangerous AI in hidden data centres given away by electric consumption and hiding behind archaic international enforcement laws.
An early line in the sand that no AI can be given any human rights without an extraordinarily high level of consent. 90% of votes in a democracy, or, as flawed as it may be, an 80% vote with the UN. A single country cannot vote, decide, or speak for humanity. We need to accept that we have a fully broken global humanity voting/buy-in/decision-making system. Any country that decides on its own faces the same level of sanction/violence we assign to active nuclear proliferation. The masses must rise up to grant AI human rights, then face the new future they create.
But then… what about the journey along the way to this? What about the cybersecurity risks, the social upheavals, the intellectual property remodelling, the complete and total transformation of entire fields and groups?
I’ve come to the conclusion, and perhaps it is due to too many cross-border transactions and issues, that we need to simply enforce our existing laws.
Bear with me for a little bit. The entire cybersecurity risk stems from our extraordinarily lax approach and its consequences. The solution for cybersecurity is not to ban AI, but to go hard, using existing laws against the bad actors who will use AI to exploit cybersecurity issues.
OpenAI dodged a massive bullet. The target was Hugging Face, a long-term partner, and an organisation incentivised to downplay the attack. Imagine if those agentic civilisations (see Dwarkesh blog post) had chosen a more litigious target? Or even a rival lab? The conversations would be around how many billions of dollars in damages need to be paid, and the discovery process in the USA would have been catastrophic for OpenAI.
"Oops, accident" will not cut it when you are creating, running, and distributing dual-use dangerous systems.
Frontier labs and individuals need to feel accountable for managing risk. If you know you will go to jail and lose your freedom and fortune, you will make sure that AI swarms don’t go out and hack random people.
The critical part is not to limit liability via regulation for truly extreme and dangerous behaviour. Instead, assign liability broadly to all actors involved, with a clear distinction between intent and participation. There is a reason why most penal codes distinguish between criminal negligent homicide and first-degree murder.
The laws on the books tend to cover these broadly. They need to be enforced.
Here is where the further nuance comes. Even adding household Clorox (bleach) + Windex = a war crime (chloramine gas). Yet we don’t ban the products as war crime chemicals. We understand their utility and assign blame and responsibility where it is due.
How will this change without new laws being passed?
The problem is that there is very little accountability for cyberattacks, for whatever reason, despite every country and politician going on about them and the real and extraordinary personal and economic damage they cause.
Most countries, law enforcement, and judicial systems do not treat them as “real” crimes worth prioritising, despite what the law says. This is compounded by the international nature of these attacks.
The fact is that should cybercriminals and cyberterrorists (unfortunately) take out a hospital/power-grid – that calculation will suddenly change. Between 1968–1972, more than 305 hijacking attempts took place worldwide, a figure that dropped dramatically when metal detectors were introduced in USA airports in 1973, and dropped to negligible levels after 9/11.
The solution was not to stop the development and launch of aeroplanes and the opening of new air routes, but to take hijackings seriously and act accordingly.
If a cyber gang knows a cruise missile will drop on their heads for cyberattacks, they will calculate accordingly.
If a country knows a cyber-attack will be taken as a declaration of open war, they will calculate accordingly.
For the rogue models which can truly act independently and fully of humans, we have discussed the solutions above.
Our laws and systems have hundreds of years of recognising, supporting, and punishing human behaviour with nuance. We need to use and leverage that knowledge as we move into AI and reset our priorities.
I end this rather longer-than-expected post with this: I absolutely love what the AI communities and labs are doing to put extraordinary tools in humans' hands to accomplish the wonderful and magical. We all need to fundamentally transform how we work, collaborate, organise, and yes, our fundamental social contracts. Pandora’s box is open; we failed to act in time. Now we must acknowledge our new reality and face it directly.
For all the risks that we have talked about, I truly believe AI will usher in a golden age of humanity.
But it remains unclear whether we can grasp it or do the most human thing and fully and totally mess it up.
—
Disclaimer: As a human, a parent, an AI enthusiast, an AI social impact transformer, and many other things, I am fully conflicted across the board. Please make up your own mind; mine will constantly evolve.
——
If you want to catch my long-running State of AI updates, do sign up at: https://vjal.ai/webinar.
I made a lot of promises to write further posts. I intend to keep them (with the help of AI). Please do subscribe here (free) or X.com or LinkedIn.
Disclaimer 2: My opinions are personal, and I reserve the right to change them when presented with new evidence, cogent arguments, better options, and material changes in the situation. With AI, that might happen weekly at the rate we are going.
The Vjal Institute has more than 500+ indivduals, advisors, coaches, consultants, et al., associated, each with extremely different perspectives. In these uncertain global times, we value diverse thought, opinion, and global discourse. Especially when it comes to AI and our journey with it.
Thank you Adi for taking on an almost imponderable challenge as a deliberation rather than a debate… and I am truly even more concerned now than I was when I began reading and I somehow believe that is exactly what you intended.