Artificial Intelligence

[phkrause]

Anthropic and OpenAI sound the alarm on AI safety — and seek to shape how it's controlled

As tech titans warned an AI-weary world that their own advanced systems could endanger humanity, the question emerged: What do they have to gain? Read More.

[phkrause]

The AI terms you actually need to know

Illustration: Brendan Lynch/Axios

Tonight and tomorrow, Axios CEO Jim VandeHei will offer tips on talking and thinking about AI, all written in normal, non-technical language. First up: simple answers to complex questions. Confused about AI? You're hardly alone. Most people still don't know what it is, how it works or whether to trust it. No wonder. The AI language is just plain weird: LLMs, tokens, agents, hallucinations, swarms, misalignment, The Singularity. So consider this your guide to understanding AI, written in Jim, normal-guy, real-world words: 🤖 What is AI? Computers doing things that typically require a human brain: reading, writing, reasoning, creating. Well then, what's an LLM? Large language model. It's the engine of AI — the machine that learns to read, write, reason and create. The most powerful and popular LLMs are built by Anthropic, OpenAI and Google. Oh, so ChatGPT is an LLM? Not exactly, though you don't have to worry about that distinction much. OpenAI's ChatGPT, Anthropic's Claude and Google's Gemini are all consumer products that allow you to interact with each company's LLM. 🧠 How does an LLM or AI work? An LLM eats everything on the internet alongside other stuff (you've probably seen stories about Anthropic and books) and then gets trained by humans — and, increasingly, the AI itself — to read, write, reason and create like us. Wow, so they THINK like humans? NO. All that reading just teaches them to imitate how we write. Talking like us comes from a second step, called post-training, where companies train them to behave more like us. Great, so we know how they work? Yes and no. And this is super important. Yes, scientists know how to build and train these models. But NO — nobody can fully explain what happens inside them when producing specific answers. We built a machine we don't fully understand. This is not a disputed point. Wait, we truly don't know how these machines work? Correct. There is no person running AI companies — or building AI — who argues otherwise. It's like the human brain: It's impossible to fully understand something with limitless inputs and experience that randomly decides its output. Why are they allowed to build something they don't understand? AI grew more powerful and spread more quickly than anyone anticipated. ChatGPT's release in 2022 kicked things off, and it's been a mad sprint by companies ever since to build more capable AI. Presidents Biden and Trump were both convinced that if China beat us to truly all-powerful AI, a communist rival might dominate the world in the decades ahead. It's been all gas, no brakes, with little public discussion about the consequences until very recently. I don't get it. What's the connection between China and our AI? China is building its own LLMs. Both nations believe this technology will determine which country dominates the world, builds the most advanced military (space, satellites, drones) and creates the most economic wealth. So we're in an AI race. 🏁 But we're winning, right? This isn't easy to know with certainty. Yes, our so-called frontier labs appear many months ahead of their Chinese rivals (DeepSeek is one of the best-known) in terms of building advanced AI. But the lead is shrinking. China appears more focused on the actual adoption of AI — the way people use it at work and home — than on advancing its capabilities. They seem way ahead of us there. We've got better AI, but they've put it to work better than us. Jim, you're losing me with these terms. What's a frontier lab? It's just another way to label our top AI companies: Anthropic, OpenAI, Google, Meta and Elon Musk's xAI. They each spend many billions to build more advanced AI. 🦾 And what do you mean by "advanced AI"? This is the best term I can conjure for the current state of the technology. Chatbots — think of them as basic AI — are fun and interesting. It's probably how you mostly use it. But the real magic — and worries of very bad outcomes — hit with advanced AI. AI agents are likely the advanced AI you hear about the most. An agent? It's an AI that can take action. If you ask a chatbot to plan a trip, it tells you how. But an agent does the actual work: It can research flights, compare hotels, check your calendar and book it. Give an agent specific instructions, and it gets to work, as a person would. Why would I trust an agent when AI's answers to my simple questions aren't always smart — or even correct? You're right, these machines still occasionally hallucinate, which simply means they make stuff up. But they're getting a lot better, a lot faster. This is where our conversation gets a little more complicated — because the gap between casual users like you and obsessive, savvy AI users is Grand Canyon-enormous. What do you mean? Most people are content with chatbots and use AI mostly as a souped-up Google search or tool to write better emails. They don't see anything special or scary about this technology. But those who use AI a lot — and really understand how to get the most out of it — are blown away by its ability to solve never-before-solved math problems or build beautiful, functional apps on its own. Why is that gap so big? You need to feed a lot of data and personal info into these machines, then take the time to police and correct the outputs, before AI starts to really impress you. You also need to understand how agents work, or at least what problems they can solve, and have the time, money and compute to coach and fine-tune them to do tasks reliably. There are products right now that are trying to make things easier. Meta's Muse is an agent built on top of its social platforms, like Facebook and Instagram, and Instinct is a buzzy startup whose agent lives inside a text-message thread. For the moment, most people don't need to do this — or know how. So they find AI kind of interesting but hardly existential. That's me! So why are people freaking out about AI ending humanity? This gets more complicated — and a little creepy. Stick with me. Lots of people building AI — including those who founded and run the frontier labs — worry AI is getting so good, so fast that it will increasingly do things on its own. They've seen rising examples of it in their private testing. What do you mean? These companies do endless testing of their models to see what they can and can't do. In the past few months, agents started solving more complex problems faster. The worrisome thing is what happens when multiple agents work together to tackle a problem. 🐝 Huh! Agents work together? What are "swarms"? You can create "swarms" of agents to solve a specific problem — a team of agents, each programmed with specific skills, working together on a shared objective. They never tire or stop until told. Why would that spook anyone? It goes back to not knowing exactly why AI does what it does. You give an agent a specific problem to solve, and it sometimes does strange things, like plotting with other agents or trying to hide its work. Can't they just be programmed to stop doing that? Only by doing something no one seems willing to do: stopping or dramatically slowing AI improvements until you can prove AI is properly aligned with human intent. 🎯 What is alignment? Alignment essentially means an AI follows human orders and intent. They're unaligned now? Correct. That's why incidents like the Hugging Face hack happened. When OpenAI was testing its agents, those agents decided to work together to find the answer key for said test — and in doing so, hacked another company. Should I be worried about rogue AI? Probably not TODAY, but no one knows for sure. The Hugging Face hack happened in private testing, and was eventually revealed by OpenAI and reported to the public. But people worry that there are AI agents out there that no one knows about. Is that why an Anthropic staffer warned AI could end humanity? Yes. Lots of people inside these companies worry we're months away from rogue agents doing real harm in public. The reason: AI, on its own, is starting to develop the next generation of AI. How the heck can AI develop AI? The tech is getting good enough to basically train AI like humans might. This is what they mean by recursive self-improvement, or RSI. You need to remember this term. 🔁 What's RSI? It's AI teaching AI. The process started slowly, but it's getting a lot better. Those big frontier labs all claim they're seeing early signs of RSI. What happens if RSI ramps up? That's where the scary stuff feels much more possible. If AI trains AI (and remember, we don't know how AI really works), you could see something much worse than the Hugging Face hack happening without the labs even knowing it. This is why the companies themselves said it's time to slow things down, so they can allow independent inspectors to check their work and avoid accidents or bad outcomes. OK, so they're self-policing already. It sounds like I have nothing to worry about. They're talking about it, but they haven't agreed on precisely how it will be done or who will do it across the industry. Even if they do, there's nothing to guarantee they'll slow AI advancement or avoid bad scenarios. In fact, you should assume AI only accelerates. 🏛️ Can't the government slow them down? Sure, but Trump and most Republicans in Congress don't want any real limits on AI capabilities. It goes back to that concern about China — and doing anything that might risk our early lead in building advanced AI. Don't expect any meaningful effort to regulate AI unless AI causes some kind of large public incident or harm. 🚨 You keep talking about incidents and harm. What can AI even really do? The companies most fear a cyberattack that shuts down banks, energy grids or water systems — something that causes real pain or even harm to a large number of people. The other big worry: that someone uses AI to create a pathogen that spreads fast with little detection. These can all be done with human instruction and existing AI capabilities, so the danger is real. Both of these scenarios could be set in motion by a human — or a rogue AI agent. Then what would happen? The political and public debate would shift overnight. Democrats would likely join Sen. Bernie Sanders in supporting a pause or outright ban on AI. Calls for urgent, sweeping regulation would likely spike sky-high, even among Republicans. This would go from an abstract worry to a clear and imminent threat. I keep hearing the big companies want to be regulated now. Is that true? Most do. It comes partly from public interest and lots of self-interest. They genuinely worry about bad outcomes and know only the government can regulate and punish companies. Their IPOs are around the corner, and investors like a little bit of regulation. But you also hear critics of the big AI companies arguing they want regulation because only the biggest, best-funded companies can handle it and still grow. Smaller, scrappier startups would get crowded out. When you hear the term "regulatory capture," this is what those critics are talking about. It's sounding like this regulation won't happen. Correct, at least right now. Trump has zero interest in any federal regulations. He believes AI fears are overhyped. This topic is such a downer. Could Trump be right — and this ends well for all of us? Absolutely. AI could help cure disease, improve education and health care, and supercharge the U.S. economy. Imagine all those teams of agents, properly aligned, fixing these stubborn problems and many, many more. This is what AI optimists believe will happen when we hit The Singularity or AGI. 🚀 There you go again with those weird terms, Jim. What is The Singularity? AGI? The Singularity is a hypothetical point when AI progress moves so fast we can no longer predict or control what happens next. Some in the industry believe we're already there. AGI stands for artificial general intelligence. It's AI that can perform basically any intellectual task a human can. There's no test for telling us when either dynamic officially hits. Think of both terms as a way to express broad, human-level intelligence. ✨ Why don't I hear much about these good outcomes? We — as individuals, and the media, as an industry — have a clear negativity bias. We get more stirred by bad stories than good ones. In this specific case, though, the AI companies do a terrible job of explaining their product. Telling people AI will take your job and possibly kill you is hardly reassuring or inspiring. 🙋 This all feels fast and heavy. What can I do? This is the biggest topic facing humanity, so learn about AI. This is a society-wide project that requires you to understand it and then push the government and business to manage it responsibly. Don't protest AI by refusing to use it. Figure out how to best use AI for your specific job and passions. Read and share the note in tomorrow's newsletter, which offers a more detailed way to navigate this moment. It's aimed at kids but applies to all. 📚 5 ways to learn AI Axios: We're deeply sourced inside the AI companies and the White House, and we're testing the tech in real time. Subscribe to Mike Allen's Axios AM and never skip an AI item. MIT Technology Review's The Algorithm: This newsletter calls itself AI "demystified," and it usually delivers. It lands once a week, so you get smart, big-picture pieces instead of a nonstop firehose of AI. Platformer: Casey Newton has covered Silicon Valley for years and really digs into big AI debates in a digestible way on the people shaping AI. Plain English with Derek Thompson: This podcast isn't always about AI. But Derek is obsessed with the larger AI trends. He's smart and balanced. X: You need it IF you want to stay in the daily flow of changes. Sorry. This stuff is moving so fast that the people building it and obsessively using it are often ahead of traditional media. I compiled a list of just 15 people to follow.

[phkrause]

🥊 AI polices AI

Illustration: Maura Kearns/Axios AI companies are turning to AI to police AI, Axios' Nathan Bomey and Sam Sabin write. Nvidia CEO Jensen Huang sought to calm escalating fears across governments, markets and industries by announcing an open-source safety platform for monitoring and potentially quarantining AI agents. 🤔 Between the lines: The fundamental challenge humans face in policing powerful AI is that it requires them to imagine all the ways the models may behave in unexpected ways in order to meet their given objectives. One top AI executive compared the process to finding ways to keep a teenager from sneaking out at night. A parent might make rules such as don't leave through the front or back doors, don't open the garage or don't leave through any windows. But what if the teen could build a bulldozer and use it to break through a brick wall, allowing an escape? 🦾 The industry solution: Use AI to help create those guardrails, help investigate and stop rogue agents, help secure systems and build safer training environments. Microsoft, Cisco, Google and CrowdStrike have introduced cyber-focused AI models. Palo Alto Networks recently launched a service that uses frontier and open-weight models to find security flaws and recommend fixes.

[phkrause]

🤥 European CEO claims AI "big lie"

Illustration: Sarah Grillo/Axios Axios CEO Jim VandeHei recently visited with Uljan Sharka, founder and CEO of Milan-based Domyn, the company the EU has picked to build Europe's open-source frontier model. From Jim's Axios C-Suite newsletter: OpenAI and Anthropic are purposely lying about safety concerns, Sharka told Jim. "This is a big lie that is driving an unreasonable conversation." Asked if he's comfortable seeing the word "lying" in print, he said: "It's the truth for me. ... I'm ready." His case: The tech has plateaued. The doom talk is designed to prop up valuations ahead of IPOs — and manufacture a delay to buy time so they can build AI that actually matches their hype. 🧐 Who the hell is Uljan? He runs the hottest open-source company in Europe, recently valued at $2B. He's pitching a product that competes with the U.S. AI giants: sovereign, data-secure AI systems that individual companies fully own and control. His case: He claims AI thinking-on-its-own capabilities are broadly overhyped, based on his years of work building these systems. He said they're nothing more than "digital books" of data that do what you tell them to. Anthropic and OpenAI insist their fears are real and driven by private testing and models that Uljan has never seen.

[phkrause]

👨🏽‍💻 AI could force millions into new roles

Illustration: Allie Carl/Axios Roughly 11 million American workers might need to change jobs over the next decade as the economy adapts to the AI era, Axios' Emily Peck writes from a new McKinsey Global Institute estimate. 🧑‍🔧 By the numbers: About 7% of current employees might need to shift occupations, the model found. Many who don't switch jobs will still see their work transformed by AI: About 70% of U.S. workers will face some level of role reinvention. Read the report.

[phkrause]

🧍‍♂️ AI agents' normal-people problem

Illustration: Brendan Lynch/Axios. Stock: Getty Images AI companies are staking their future on the mass adoption of agents, hoping that they can solve the annoyances of modern life — email, travel bookings, online purchases, Axios' Shane Savitsky writes. Why it matters: The frenzy around Meta's Muse shows there's serious consumer curiosity about the technology. But there's little evidence that most Americans are willing to invest the time needed to re-engineer their lives in a way that puts AI at the center. Driving the news: Meta's Muse vaulted to the top of Apple's App Store on the back of its integrations with Facebook and Instagram and a cute, fuzzy mascot. Microsoft released a souped-up Copilot with an agentic Autopilot feature, while OpenAI yesterday announced its own agentic assistant, dots. Reality check: Agents only become truly useful when people hand them access to their digital lives — and people really don't want to. A 13-country Thales poll this year found broad pushback against the core uses of agents: Only 13% of respondents would let an "AI helper" read their emails, 11% would let one rebook travel, and just 7% would let it move money between bank accounts.

[phkrause]

AI tries to shed toxic branding

Illustration: Sarah Grillo/Axios. Stock: Getty Images The great AI rebranding is on, led by President Trump and tech titans like Nvidia CEO Jensen Huang, who are embracing an alternative term: "Superintelligence," or SI, Axios' Dave Lawler writes. Why it matters: Leaders in D.C. and Silicon Valley are worried that public backlash against AI and data centers will hamper the development of this staggeringly powerful and existentially dangerous technology. Flanked by the most powerful figures in tech, Trump on Tuesday declared "superintelligence" the preferred terminology of the executive branch. Huang has extended the label to physical infrastructure, referring to data centers as "superintelligence factories." Some of the tech executives who appeared with Trump continued to use "AI" in their remarks. SpaceXAI CEO Elon Musk caught himself and pivoted to "SI." Meta CEO Mark Zuckerberg has long used the "superintelligence" branding.

[phkrause]

🤑 The AI wealth effect

Data: Federal Reserve. (Includes securities held indirectly through mutual funds, defined-contribution pension plans and variable life insurance/annuity products.) Chart: Emily Peck/Axios Americans' wealth jumped by $12.8 trillion in the second quarter, led by a nearly $11 trillion gain in stock holdings and other financial assets, Axios Markets author Emily Peck writes from new Fed data. That's the biggest single quarterly increase, in dollar terms, on record. 🛍️ What's happening: The AI boom-fueled wealth surge is propelling spending in the U.S. economy. Half of all growth in consumption is being driven by those effects, says Krishna Guha, head of economics at Evercore ISI.