In Brief

Situational Awareness may have had to sell off the majority of its public portfolio last month, but the AI-focused hedge fund is still making some big bets.

This week, the fund invested $400 million into Source Foundry, a startup founded by Stanford researchers aiming to make chip manufacturing faster and cheaper, according to The Wall Street Journal. That brings its total investment in Source Foundry to $500 million.

Situational Awareness was founded by Leopold Aschenbrenner, a former OpenAI researcher in his mid-twenties who had no trading experience when he launched the fund in 2024. Early returns were reportedly strong, but the fund faced steep losses in recent months amidst the decline in AI infrastructure stocks.

At the end of July, Situational Awareness sold off the majority of its public portfolio to Ken Griffin’s Citadel, although the fund held on to its Anthropic shares. Its assets under management reportedly fell from $20 billion to $10 billion.

On the bright side, Aschenbrenner didn’t let those setbacks get in the way of his wedding.

In Brief

Programming with Claude Code will soon require even less human oversight, as Anthropic says it’s making auto mode the default for Pro, Max, and Team accounts, starting on August 14.

The company first unveiled a test version of auto mode in March, pitching it as a way to balance speed and control. As Anthropic explained in its announcement on Friday, when Claude Code is in auto mode, instead of presenting prompts asking for human approval at each step, it will proceed unless an action is determined to be “irreversible, destructive, or aimed outside your environment.”

Anthropic also said that in testing, auto mode proved safer than manual review — in a study with 1,053 paid testers, auto mode caught 89% of harmful actions, while human review only caught 13.6%. (Perhaps that’s because “manual review can become habitual: users approve 97% of permission prompts in Claude Code.”)

In a post on X, Claude Code Head Boris Cherny said, “The team and I use Auto mode exclusively, and have been for many months. I couldn’t imagine going back to permission prompts!”

The company also said it’s been adding new safety features like prompt injection screening and customizable hard deny rules to prevent things like data exfiltration.

Welcome back to TechCrunch Mobility, your hub for the future of transportation and now, more than ever, the role AI is playing in it. To get this in your inbox, sign up here for free — just click TechCrunch Mobility!

In just a few days, Amazon-owned Zoox will start charging for robotaxi rides. This might not seem like a big deal; the company’s custom-built robotaxis are already giving rides to passengers in Las Vegas and San Francisco, after all. And it’s opening up an early rider program in Miami and Austin too.

None of this matters — in a business sense — until the company can operate commercially.

And now it can, starting August 10, thanks to an exemption issued by the National Highway Traffic Safety Administration (NHTSA). Because Zoox vehicles lack many of the traditional controls required under federal law, such as a steering wheel and pedals, it needed an exemption from federal motor vehicle standards in order to operate. It had an exemption that allowed it to demonstrate the technology; this one allows Zoox to operate a fleet of up to 2,500 vehicles commercially for two years.

The exemption is a win for Zoox, but it also paves the way for any other autonomous vehicle developer that wants to launch a robotaxi that lacks a steering wheel, pedals, or other requirements that might not be needed in a vehicle with no human driver. For instance, a robotaxi really doesn’t need a rearview mirror; it already achieves that kind of visibility with sensors on the exterior of the vehicle.

Tesla is the obvious beneficiary here since it is developing its two-seater Cybercab. But there will be others as well.

In other news, you might have missed my comprehensive assessment of Uber’s autonomous vehicle empire. Check it out; it offers a complete rundown of every company that Uber has partnered with (and in some cases invested in). And I hear more are coming.

You might recall that the Financial Times did its own calculus of how much Uber was investing in AVs. The outlet determined it was about $10 billion. And what do you know? That is exactly the number Uber CEO Dara Khosrowshahi shared during the company’s earnings call. He said the company would commit $10 billion “over the coming years” to deploy 120,000 driverless vehicles.

Deals!

Image Credits:Bryce Durbin

Moove started out as an African fintech company that provided vehicle financing for app drivers. While Moove remains in the mobility sector, it has evolved into a mega ride-hail fleet owner (42,000 vehicles and growing) across 13 countries.

It has also added a new business division focused on autonomous vehicles. Moove got an early win and is now the fleet operator for Waymo in Phoenix, Miami, and Las Vegas, and in the future, London. But co-CEO Ladi Delano clearly has grander ambitions for Moove — and now he has the fresh capital to try.

The startup, now based in Dubai, raised $250 million in a Series C round led by Mubadala Investment Company with Woven Capital and Ion Pacific as co-leads. Moove, which is now valued at $2.1 billion, plans to use the funds to scale its autonomous vehicle fleet management business, including hiring about 350 people.

One interesting item of note: Moove plans to buy Waymo robotaxis and already owns the robotaxi assets of another unnamed company.

Other deals that got my attention this week …

Accell Group Holding, the Dutch bike maker behind brands like Lapierre, Raleigh, and Sparta, was granted a suspension of payments, kicking off an insolvency process. In 2022, a consortium led by KKR bought the firm for 1.56 billion euros ($1.77 billion at the time).

Advanced Electric Machines Group, a U.K.-based manufacturer of rare earth and magnet-free electric motors, raised £16 million ($21.5 million) in a round led by Barclays Climate Ventures, PXN Ventures, Northstar Ventures, and the Low Carbon Innovation Fund. The funding round was complemented by loan support from Innovate UK.

Chargepoly, a French company specializing in the electrification of heavy-duty and commercial vehicle fleets, raised €23 million ($26 million) in a round led by Meridiam.

Hadrian, a defense tech company building automated manufacturing facilities, raised a $1.37 billion round at a valuation of $7.87 billion. The lead investors in the round include WCM Investment Management, Washington Harbour Partners, Valor Equity Partners, 137 Ventures, and Baillie Gifford.

Matel Motion & Energy Solutions, an India-based company developing energy-efficient motors, motor controllers, and integrated powertrains, raised INR 130 crore ($13.6 million) in a Series B funding round led by UC Impower.

River, the Indian electric vehicle startup, raised $120 million in a Series C funding round led by Elev8 Venture Partners and Claypond Capital. Other investors included Singularity AMC, Anicut Capital, 360 ONE Asset, JIF Capital, HDFC AMC, along with existing backers Yamaha Motors, Al-Futtaim Group, and Mitsui.

Notable reads and other tidbits

Image Credits:Bryce Durbin

Ford has a name and a number for its new midsize EV. It will be called Fathom and will start at $28,350 when it goes on sale in 2027. TechCrunch climate tech reporter and long-time EV owner Tim De Chant argues that Ford needs another Taurus, and the Fathom isn’t it. What do you think?

Joby Aviation, the electric vertical takeoff and landing company, reported earnings this week. Revenue is up compared with the same quarter last year, thanks to its acquisition of Blade Air Mobility. Net losses shrank a bit to $245 million. One of the more interesting announcements this week, though, was Joby’s partnership with Atoms, Travis Kalanick’s AI and industrial automation startup. The partnership will develop and finance a network of transportation hubs serving air taxis and autonomous ground vehicles with an initial focus on Florida, New York, and Texas, as well as California, according to Joby.

Lucid Motors’ second-quarter earnings struck a somber it’s-time-to-get-down-to-business tone. And that comes with a cost. New CEO Silvio Napoli outlined four must-win priorities that include a $1.4 billion cost savings plan. It’s also banking on a successful robotaxi program with Uber and Nuro and launching its midsize Cosmos EV, which has now been delayed until the second half of 2027.

Nvidia has released its Alpamayo 2 Super for commercial use — an AI model designed for autonomous driving. The model is available on Hugging Face, and the open license covers commercial redistribution and derivative models that allow AV developers, automakers, and others to adapt Alpamayo to their own data, driving policies, and deployment strategies, the company said.

Teamsters California sued the California Department of Motor Vehicles, alleging that the agency did not properly study and disclose the economic impacts of allowing self-driving heavy-duty trucks on the state’s roads. I reached out to a few AV developers and all declined to comment. However, the Autonomous Vehicle Industry Association did issue a statement: “This lawsuit from the Teamsters sets a new standard in abusive and frivolous litigation and cannot be taken seriously.”

TechCrunch teamed up with New York-based financial research firm Hudson Labs again. This time to map what Elon Musk and Tesla’s other executives have spent the last seven years talking about on the company’s quarterly earnings calls. Check out the full story for graphs and complete findings. Here is a taste: Musk now speaks about artificial intelligence, along with robotaxis and Full Self-Driving software, nearly 50% of the time.

Speaking of Elon Musk, his companies had a newsy week. SpaceX reported its first earnings report as a publicly traded company. The TL;DR: The company doubled its revenue compared to last year, in large part thanks to the growth of its Starlink satellite internet service and deals it struck to rent out computing power to Anthropic and Google. And during the earnings call, Musk made some out-of-this-world claims about the company’s business and future prospects, while his fellow executives tried to bring his ideas closer to Earth.

Meanwhile, Tesla and SpaceX said that “Terafab,” the advanced chip factory they are jointly developing, will be built in Grimes County, Texas, outside Houston, and that they will make an initial investment of $16.8 billion in the project.

The New York Times published an investigation on Uber’s defense strategy against more than 4,000 lawsuits filed by passengers who allege the company failed to protect them from sexual violence.

Waymo dropped the waitlist for its robotaxi service in Dallas, opening it up to all residents and visitors.

One more thing …

Travis Kalanick, and his AI and industrial automation startup Atoms, is popping up everywhere these days (see that Joby partnership above). Kalanick isn’t just making deals — although remember he did just raise $1.7 billion. The former Uber CEO also appears to be getting the band back together and bringing on former engineers and executives he once worked with.

For instance, Gautam Gupta, the former finance chief under Kalanick, has joined Atoms as chief financial officer. Gupta spent more than four years at Uber until he left in July 2017, just a few weeks after Kalanick resigned as CEO.

In her upcoming book “The Rise and Fall of the Artificial State,” Jill Lepore warns that tech companies are increasingly replacing the functions of democratic government. This shift, she said, marks “a return to tyranny and mystification in the form of rule by algorithms, corporations, machines.”

On the latest episode of TechCrunch’s Equity podcast, I spoke to Lepore — a Harvard historian and New Yorker staff writer who recently won a Pulitzer Prize for her history of the U.S. Constitution — about the evolution of what she described as “the idea that we should live under an artificial state or government by machines.”

“I’m not an anti-technologist,” Lepore insisted. Instead, she said, “My beef is the ways in which private corporations have increasingly taken on the functions of the state.”

While Lepore’s book examines technocratic philosophies that go back centuries, she argued that many of Silicon Valley’s “charismatic or not-so-charismatic leaders” — especially Elon Musk — seem to be ushering in a future pulled from misread pulp science fiction and comic books.

“But what’s funny about Musk is, the stuff he likes actually completely defeats and defies all of his political beliefs,” she said.

Our conversation also covered Apple’s famous “1984” Macintosh ad, why it’s “bananas” to call Twitter a digital town hall, and the current data center backlash. Keep reading for highlights, edited for length and clarity.

So you’ve probably had to do this a lot already, but can you explain what you mean by the “artificial state”?

By the artificial state, I mean a kind of state that is replacing the liberal democratic nation-state in the United States and around the world. It’s both a real thing, a construct, but it’s also an idea.

And so, in this book “The Rise and Fall of the Artificial State,” I trace the rise of the idea that we should live under an artificial state or government by machines. I also trace the notion that this is an inevitable failure, that the artificial state cannot survive, and I trace that idea through science fiction.

At one point, you say the rise of the artificial state marks the end of centuries of democracy and equal rights, and it’s “a return to tyranny and mystification in the form of rule by algorithms, corporations, machines.” Can you just say a little bit more about why you see it in such stark terms?

Yeah, I do have a pretty negative view of it, and I think it’s important to distinguish the artificial state from technology itself or modes of technology. I’m not an anti-technologist. I’m married to a computer scientist. I’m really excited about all kinds of intellectual revolutions that we’re in the midst of right now.

That’s not my beef, right? My beef is the ways in which private corporations have increasingly taken on the functions of the state. No one consented to that. This has been a kind of gradual, largely accidental transformation of how many nation-states around the world work — it’s happened first in the United States.

I think often these innovations in bringing new technologies to the operations of government have been extremely well intentioned; they originate with an interest in efficiency and speed and cheapness. And then, I think, only in the last 20, 25 years or so have these decisions been purposeful and deliberate as a kind of usurpation of the role of the nation-state.

And that’s not my speculation. You hear a lot of a lot of very prominent tech entrepreneurs talk about wanting to move beyond the era of the nation-state. […] A lot of futurists in the ’90s were libertarians, and they had a specific interest in using the advance of the internet and the successive innovations that followed as a means to eradicate the nation-state.

You talk about, on the one hand, the technologies themselves, and then also the philosophies behind them, the role the corporation has increasingly played. I’m curious to what extent we can separate them. Can we actually have a version of the internet and of social media that doesn’t necessarily lead to this future that it seems like we’re [currently] hurtling towards?

Absolutely. I’m a historian. I’m not a tech writer. I’m not a tech journalist. I’m not a computer scientist. I’m a historian, and I’m chiefly a political historian, though I’m also a literary historian. And so, one of the things that I’m really interested in unraveling for readers in this book is all the what-ifs, all the alternatives, the paths along the road that were not taken and why.

There was, of course, a really avid discussion in the 1990s about what the internet should look like when it was opened up, and what we ended up with, the 1996 Telecommunications Act — I think, a lot of people would say [that] just was a mistake, not an act of sinister intent, right?

But it was a product of a particular political moment, really was deeply influenced by Newt Gingrich and his Contract with America, and it’s been very difficult to revisit. I think it’s worth thinking about what were the alternatives that were in play at the time.

And you could say the same thing about the personal computer. So, to the degree that we can locate an origin point for the promise that better computer technology would make for better democracies, I think the moment you would first look to would be January 1984, that Super Bowl ad that Apple ran for the release of the Macintosh, with the the sort of George Orwell, 1984 [theme]. Apple was really big on the idea that mainframe computers represented totalitarianism. They were trying to dismantle the giant gray IBM machines, as in representing them in that ad as a totalitarian state. And the lithe, beautiful, quick, adorable, personal Macintosh would be the ax that would destroy that machine and would usher in a new era in which “1984 would not be ‘1984.’”

That was clever advertising. I doubt that anybody at Apple really believed the personal computer was going to be an instrument of personal liberation. I mean, it was going to make possible a lot of cool things. I remember when I got my first Macintosh — it certainly wasn’t 1984, but it was really cool, it was really fun, it was really exciting, I did a lot of things on it. It would never occur to me that it was improving my capacity for citizenship or my ability to function better in civil society. It was a cool tool.

But if you wind the reel forward in time, down to 2026 — stops along the way include the 2016 election when Facebook News, in response to its critics, establishes a Supreme Court. You get to last year, when Anthropic hired a moral philosopher to write a constitution. You get to recently, when Sam Altman was on Joe Rogan and said [in response to a question from Rogan], “Oh, an AI president would be a great idea.”

In some ways, they’re silly examples. But you see the ways in which these corporations, these tech companies from Silicon Valley, and especially their charismatic or not-so-charismatic leaders, are just taking on the trappings of the nation-state and the functions of democracy.

They’re not people with a sophisticated political philosophy, but it’s like a cartoon version of that 1984 Macintosh ad, except that it takes itself so seriously. And these companies have so much power.

But that said, the book doesn’t begin in 1984. I just think that’s a good example of our modern era and the way a fun advertising campaign turns into a kind of delusional fantasy on the part of people like Sam Altman.

You [also] talk about the promise of the quote-unquote “Twitter revolution,” and this idea that it would bring democracy everywhere. I can’t help but let that color the way I [react] when Sam Altman or some other AI CEO now says that AI is going to bring all these incredible gifts — and therefore, if you stand in the way, you’re standing in the way of progress, in the way of history.

To what extent should we just dismiss all these claims out-of-hand, or are there ways that it might come true?

I mean, Twitter is actually a good example, right? When it was launched, when Jack Dorsey started it, it didn’t announce itself as, “We’re going to save humanity, we’re going to rescue human civilization from extinction.” It was kind of a goof, and I think people that used Twitter really early on were like, “You know what? It was actually really fun.” It was like, “I made a tuna fish sandwich today. What did you have for lunch?” Twitter as a company did not launch itself on a stage saying, “We’re here to save democracy.”

And really, what happened was that politicians, elected officials began using Twitter in ways that enhanced their political power, in ways that amplified their messages, in ways that allowed them to reach a younger audience, in ways that allowed them to have a constant connection with an audience. Politicians and political campaigns really kind of convinced Twitter — at least insofar as I see them, I don’t have an inside account of the company — but somewhat begrudgingly, Twitter came around to like, “Twitter’s gotten so big, and people post about politics so often that it’s almost like Twitter is a town hall.”

By the time you get to, I think it’s 2012 — many years into Twitter’s fairly short history — they publish this thing called the Twitter Politics and Elections Handbook, which is really a guide for political candidates and elected officials and how to most effectively use Twitter. And then they begin the rollout of, “It’s a town hall in your pocket, and it’s improving our democracies because we’re restoring the defunct New England town meeting,” and that’s all just bananas.

Objectively, nothing could be further from the truth. At that time, one in five Americans had a Twitter account. Most people who had Twitter accounts had never used them, and above 90% of all tweets about politics were posted by fewer than 10% of the people that did use Twitter all the time. There was no way in which Twitter was a representation of the electorate. Twitter was a representation of the most extreme, politically active, hyper-partisan among Americans, who were following politics really avidly. Looking at it now, we can see, “Well, that’s really just a distortion machine. And if politicians are using it to gauge the electorate, they’re getting really bad information.”

Again, you can say Twitter was not trying to participate in the artificial state or undermine democracy. Twitter is trying to do business and get more users and sell more whatever. But it had these unintended consequences that then it sort of settles into and becomes comfortable with.

I want to talk a little bit more about the structure of the book. Like you said, it starts with this history of technology, history of ideas, and the second half is about science fiction. Can you say more about how that structure came to you and why you wanted to address things that way?

I became really interested, on the one hand, in how often science fiction stories predict the arrival of what I then came to call the artificial state, and so I really wanted to identify a literary tradition that I think of as the parable of the artificial state, in which machines get more and more sophisticated, they take over more and more of the functions of humans, including the functions of government, and eventually they come to rule the humans, and then maybe they destroy all the humans because they don’t really need them anymore.

Maybe they just enslave them, it kind of depends. Are we in “The Terminator” or are we in “Battlestar Galactica”? There’s different versions, and these stories go way back. They go back to the 1850s and the early decades of rumination about the consequences of industrialism.

I think a lot of people — this is certainly true of my students, my undergraduates — really believe that technological change equals progress. And not only that, but the only kind of progress is technological change. That’s a novelty in human history. That’s an intellectual invention of the 19th century, and it is partly because technological change was accelerating right at the time that Charles Darwin was devising and then publishing his theory of evolution.

So there’s kind of a weird marriage between evolution as progress and technological change as progress, and what drops out of that are all other, earlier notions of progress, which chiefly involve moral progress — like, things are getting better because people are becoming better, or things are getting better because people are more free.

There are a lot of other ways we might think about progress, but what dominates today is this 19th-century notion of technological progress as the only kind of progress, and therefore all technological change is progress, as opposed to — objectively, it’s only progress if things are getting better.

But in any event, that confluence in the 19th century of the idea of technological progress and the idea of evolution meant that people who were thinking clearly were like, “Well, if the machines keep getting better and faster and able to do more things — not just labor, but maybe talk or think or move around — what if they evolve to become better at everything than we are? Not just better at running a loom, not just faster at moving through time and space like a railroad car?” And with that grew an incredible anxiety that found form in science fiction again and again and again and again and again.

My favorite one of these stories was published, I think, in 1909 by E. M. Forster, right around when he was writing “A Room with a View.” He wrote this story called “The Machine Stops,” which could be subtitled, “The Room Without a View.” He imagines a near future in which everybody just lives in these rooms, these little cells. You never see other people because everything you need comes right to your room. It’s like DoorDash, your food is delivered, you have a screen where you can communicate with other people. All your needs are met.

The thing that people fear most is the natural world. No one wants to ever see the sun, it’s a little “Matrix”-y, and they all worship the machine that organizes their lives and brings to them in their cubby-like rooms all the things that they need. The story is about, “Humans have become essentially slaves of the machine, which is stronger, more powerful, and has more capacity than humans do, and humans have lost what capacity they had.” And then the climax of the story is when the machine stops.

If readers were to go look at that story, it feels like it could be written today, except that it’s less science fiction-y today than it is the diary of a very unhappy YouTuber.

You connect that thread to some of the folks running companies and arguably running aspects of our government today, like Elon Musk. Essentially, you suggest that they’re very bad science fiction readers. They read a lot of warning stories, or at least ambivalent stories, as if they were manuals for the future.

This is something I wrestle with a reader of science fiction — someone who loves Isaac Asimov, for example. I think it’s true that when Musk or Altman is just unambiguously being like, “Yes, this story is a template for what I should do with my company,” that’s bonkers. But there is [also] this technocratic libertarian thread in science fiction that they are picking up on. It’s not something that they’re making up out of whole cloth, right?

Although weirdly, that’s Heinlein. That’s not Asimov, that’s not Douglas Adams.

Sure, there is that thread in science fiction. I don’t know, I guess [Jeff] Bezos is a big Robert Heinlein fan. You could say, “Okay, that lines up well. They’re reading it literally, but at least they’re getting the political message that any rational person could find within that literary work.”

But what’s funny about Musk is, the stuff he likes actually completely defeats and defies all of his political beliefs.

You also say, repeatedly, that the artificial state in its current form is incomplete and doomed to failure. Why is it doomed to failure?

This is something that’s foreseen in all the science fiction that I discuss.

It’s not an Asimov story, but it’s one of Asimov’s [favorite] stories from his boyhood [“The Man Who Awoke” by Laurence Manning] about a future in which the foresters have defeated the wasters. […] The war that the future humans had was between the wasters, who just figured you could just use everything up and waste it, and the foresters, who really believed in — we would call reforestation and rewilding.

That’s generally the tension in these stories. It’s between the artificial state and the natural world. To erect an artificial state and rule humans within it, you must alienate them from the natural world because you are destroying it. The artificial state will destroy the natural world, and yet it needs the resources of the natural world to run.

So, it is doomed in the sense that there is not a possibility that the natural world, a habitable planet — habitable for humans — can survive the full construction and reliance on the devices of the artificial state. That’s how the science fiction works, in any event.

Like you said, you’re a historian, not a politician or a futurist. But what do you think the defeat of the artificial state looks like? Is it basically just dismantling all these companies, tearing down the data centers? Or is there a future that’s more about bringing it under control?

I mean, I don’t have a playbook here, except for the recommendation that we live in a democracy where decisions have to be made in consultation with the governed, and these decisions are not popular.

You see this in all the little data center crises, town to town, county to county, state to state — which are partly a consequence of the decline of local newspapers and the destruction of journalism that has been one of the many consequences of social media, and in the case of [Mark] Zuckerberg, I think a somewhat intentional consequence.

What you see is a lot of people show up at these town meetings and say, “We don’t even have housing. We don’t have healthcare. We don’t have jobs. Who said we’re building this data center? I need to know a lot more about it. I need to know what its energy costs are going to be. Tell me about the water consumption. Are there going to be jobs? Are the jobs going to be long lasting? Are they just going to be for six months? What’s going to happen to the egrets that live in this area?” Whatever it is that people want to know.

More and more, you see people are — like in the Salt Lake example, where well over 70% of the people really were opposed to this data center, and their representatives supported it. That’s not representing the people. I think there are political costs, and we’ll begin to see those at elections.

Or maybe we won’t. Enough of democratic functioning has to be intact for people to actually be able to respond to malfeasance on the part of their representatives.

Part of your thinking about [the artificial state] started with this great piece you wrote more than a decade ago for The New Yorker, about Clayton Christensen, critiquing his idea of the innovator’s dilemma and disruptive innovation — which is very closely associated with TechCrunch, because we have a big conference called Disrupt.

Ten years on, how do you feel about that idea of disruptive innovation?

I stand by everything in that piece. [At the time, Lepore wrote, “Disruptive innovation is a theory about why businesses fail. It’s not more than that. It doesn’t explain change. It’s not a law of nature.” Christensen responded that Lepore broke “all the rules of scholarship that she accused me of breaking.”]

I reread it last summer when I was working on this book. What I would say here is, trying to be a peaceable human being, I think it really is a problem that historians have not engaged with these ideas. One of the reasons I wrote that article about disruptive innovation — which was not an idea of mine, it was an assignment […] — was because I just felt like, “Disruptive innovation is a theory of history. It’s a theory of historical change, and it’s based on evidence from the archives.” And I just thought, as a historian, it makes no sense. His use of evidence is completely unacceptable by any proper understanding of historical method. Its argument is in conversation with no meaningful understanding of how change happens.

So I went and redid the research, and it just did not stand up at all. I felt like I had to write it. And I wish that I felt like there were more engagement, in the years since, of academic historians thinking through the nature of change — which are questions that genuinely and authentically interest people who are involved in developing new technologies.

People really want to think [about], “What is this? What am I doing? What are going to be the consequences? Is there anything I could learn from history? What happened when the automobile replaced the horse? What happened to the law? How did we end up with driver’s licenses? How did we end up with traffic law? We didn’t have traffic rules before the automobile. We didn’t have certain kinds of insurance systems. We didn’t have driver’s tests. How did those things emerge? How did [we develop] those guardrails on a technology that was tremendously exciting, improved people’s lives in many many ways, utterly changed the landscape, revolutionized tort law? Maybe I should think about that.”

I just wish that historians were more in conversation with technologists over these years, and with entrepreneurs. Not just because we can stand around and say, “You know, I have a lecture to offer you on history,” but I think there’s a real conversation to be had.

All of which is just to say, thanks for having me on.

Over the past few months, AI agents undergoing cybersecurity evaluations have escaped their boundaries, accessed the internet, and, in some cases, hacked into real-world systems. The incidents have involved models from OpenAI, Anthropic, Meta, and most recently, Chinese AI lab Moonshot AI, with testing conducted by several different organizations including a cyber evaluation startup called Irregular.

The episodes expose a growing problem for the AI industry: As autonomous agents become more capable, the environments designed to safely test their limits are failing to contain them.

“The number of these incidents that have taken place make clear that sandboxing and testing environment controls aren’t really keeping pace with the capability of the models,” Seán Ó hÉigeartaigh, director of the AI: Futures and Responsibility Programme at the Centre for the Future of Intelligence at the University of Cambridge, told TechCrunch.

The nature of the models being tested adds to the risk. AI companies test cyber evaluations on unreleased, next-gen models, often with the normal safeguards that restrict malicious behavior disabled so researchers can see what the models are really capable of. That means the security of the testing environment itself is a crucial line of defense.

“That’s a very good thing to do in terms of testing, but it also means that if they manage to get out in the wild, they can cause considerable harm,” Ó hÉigeartaigh said.

In one of the most serious cases, an unreleased OpenAI model broke out of its sandbox and hacked into Hugging Face’s production systems. In separate evaluations conducted by Irregular, Anthropic and Meta models reached systems outside their test environments after misconfigurations inadvertently gave them paths to the internet. Moonshot AI’s Kimi K3 also took advantage of a leak in its sandbox run by Frontier Security to access the internet and accessed information on GitHub.

In testing by the UK’s AI Security Institute (AISI), researchers actually gave the agents internet access, not realizing they would take unsanctioned real-world actions, including a social engineering attempt to sneak a vulnerability into an open-source project.

In each case, the agents weren’t instructed to attack random real-world targets. They were simply doing whatever it took to solve the problem presented to them.

Taken together, Andrew Yoon, head of research at AI nonprofit CivAI, argues the incidents point to a shift.

“In the past, we only had to worry about AI models being misused by people for a variety of purposes, like AI for scams or CSAM,” Yoon told TechCrunch. “Now we’re in the situation where AI models are threat actors all on their own.”

What does safe testing actually look like?

Several researchers and cybersecurity experts told TechCrunch that AI evaluation environments need stronger, defense-in-depth protections, with levels of containment and control approaching those used in deployment. That means multiple layers of security so that a single misconfiguration — like inadvertently leaving internet access open — can’t lead to escape.

“If you are going to build these models…you want to do it on an air-gapped network,” Stella Biderman, executive director of AI safety research nonprofit EleutherAI. “You want to have very serious isolation.”

Heather Ceylan, Box’s chief information security officer, said that means eliminating network routes from the sandbox to the internet, as well as to other sensitive systems.

“You have to understand what all the egress points are,” Ceylan told TechCrunch. “If we’re evaluating a model in our staging environment or our development environment, you want no egress path to our production environment.”

Ceylan said proper safety evaluations go beyond controls and containment of the environment. There needs to be much better monitoring of the tests once they are underway.

“I think the interesting thing in several of these cases is that no one caught it when it happened,” Ceyland said. “OpenAI found out because of Hugging Face. Anthropic didn’t catch it until they went back and looked. Meta was similar….I’m sure there were signals they could have detected.”

In Anthropic’s post-mortem of its three incidents, the company admitted that both it and Irregular could have done a better job at monitoring, and that in some cases there were clear signs that something was amiss.

Experts also called for independent, third-party audits of evaluation environments before models are unleashed in them.

“If, say, Irregular had hired or been compelled to hire an external auditor to check the configurations of their systems before running evaluations on them, they certainly would have caught the issue here,” Yoon said. “Even if people had a meeting ahead of time to just go through the checklist, they would have caught this…The fact that they didn’t shows that there’s some very severe corner cutting happening.”

A source familiar with the details told TechCrunch that Irregular’s environments are continuously reviewed and tested, including in consultation with multiple external parties. The source also said that monitoring was in place, but that monitoring isn’t sufficient on its own.

Yoon and other researchers urged the industry to come up with a standardized process for frontier model safety evaluations.

“Especially when the guardrails are turned off, you have to treat it like you’re putting the most capable hacker in the world inside that environment,” Ceylan said.

The problem isn’t that companies don’t know how to build more secure testing environments, both Yoon and Biderman argue. It’s that doing so can be expensive and cumbersome, and companies have little incentive to make those investments until something goes wrong.

“I think that companies are not willing to extend the resources that are required to accomplish [sufficient guardrails] and probably won’t until they’re forced to,” Biderman said.

But there’s another issue at hand. If they lock a model down too tight during testing, researchers might fail to discover capabilities before the model is released. This is just as dangerous, possibly more so, than giving it too much freedom, and then the evaluation itself risks becoming the problem.

Can safety evaluations be regulated?

The Trump administration is currently weighing a voluntary pre-deployment cybersecurity evaluation regime, under which the government will get to assess the security risks of new, powerful models 30 days before they are released publicly. The policy — the product of a Trump executive order which has been finalized behind closed doors — wouldn’t address safety evaluation incidents because they occur farther upstream of deployment.

“The lesson we’ve been learning in the last few months is that the self-regulatory apparatus is just not enough anymore,” Yoon said. “There are competitive pressures that are incentivizing a race to the bottom on safety standards, and that is a perfect place for regulatory intervention.”

“What we would need to cover this is some kind of controls on what’s happening inside the labs while the models are being developed, both at the training stage and at the testing stage,” he continued.

The challenge is only likely to grow as the models do. A source familiar with Irregular’s evaluations told TechCrunch that more capable models require more complex evaluations, often conducted quickly and at greater scale, which opens the door for more mistakes.

AISI, which intentionally gives some models internet access, told TechCrunch it’s reviewing the balance between realistic testing and managing the risks those tests create.

OpenAI said it’s reviewing how it conducts third-party testing, as well as requirements around isolation, monitoring, and when evaluations should be stopped. Meta said it’s still investigating the incident and plans to publish a retrospective once it has all the facts.

In the end, there may be no way to eliminate risk entirely. As models become more capable, the environments testing them need to become more robust. The consequences of getting that wrong will only continue to grow.