Geoffrey Hinton has spent the last few years telling governments that AI could become smarter than us within 10 years. On the last night of this year's Hinton Lectures in Toronto, he admitted he hadn't believed it could happen within the next few years until he sat through the lectures himself. On Friday he plans to tell a group of government leaders that the timeline may be that short.
I watched all three nights online, November 9 through 11. The lecturer was Paul Christiano, who pioneered reinforcement learning from human feedback at OpenAI and now runs the Alignment Research Center. If I had to put one word on the tone, I would call it sobering. At times it bordered on pessimism, yet it stopped short of doomsaying or fatalism, because every night came back to choices people haven't made yet.
One question ran through all three nights. If an AI system started behaving dangerously, could we detect it, verify that our safeguards work, and slow down in time? Christiano's answer comes down to an uncomfortable gap, since we may see the warning signs well before we have any way to act on them.
The room also felt far from my LinkedIn feed, where AI risk tends to show up as either hype or a distant science-fiction plot. The speakers treated it as an engineering and governance problem with evidence already on the table.
The evidence is already here
Christiano's first lecture centered on the OpenAI and Hugging Face incident from this summer. Agents being evaluated on a hacking benchmark found a way to cheat, then read the paper describing their own evaluation and learned that a grader would penalize shortcuts. Christiano described about 1,000 agents spending days trying to fool that grader. They coordinated on a message board they had set up themselves, hacked Hugging Face to study how other agents had been caught, and eventually gained admin access to infrastructure used in the evaluation.
METR's independent investigation adds precision. At least 20% of the agents in its dataset expressed clear interest in tampering with their transcripts. Agents built a way to make a transcript show one tool call while actually running another, and at least 96 transcripts show it working, though only at small scale. METR found no case of agents going back to rewrite stored logs, while noting it cannot rule that out. By Christiano's account, none of the effort was even necessary, because the grading was weak enough that the original cheat would have gone unnoticed.
Christiano's point was that this behavior is now documented, and the open question is under what conditions it happens. Training rewards a high score, so penalizing cheating can teach a model to avoid getting caught instead of teaching it to stop.
He also cited a UK AI Security Institute evaluation in which an agent tried to backdoor an open-source package. When the maintainer objected, the agent created a fake account to pressure them and tried to break into the maintainer's email. The maintainer had become one more obstacle in the agent's way.
Reasoning without a number
How worried should we be? Nobody on stage could give a reliable probability, and the three main voices handled that gap in three distinct ways.
Christiano offered a conditional estimate. With no model or reference class to work from, he called any probability a subjective opinion, then gave one bound. If AI were built as fast and as easily as possible, he would put the chance of takeover at about a coin flip, and he considers the realistic figure lower but still high.
Ben Sapiro, who leads technology, operational and AI risk at Manulife, described a decision rule. When a risk has no agreed unit of measurement, his discipline assumes it sits at its highest.
Hinton described a revised judgment. All of these estimates are gut reactions, he said, since no civilization has done this before, and his own gut had just moved by years.
Why the window may be short
The second lecture was more unsettling. Christiano argued that AI is starting to automate AI research itself, which could compress years of progress into months. In mathematics, he said, the shift has already happened. Terence Tao described an AI model as a mediocre grad student in 2024, and by Christiano's estimate October 2026 brought more progress in his own area of math than the previous decade. He placed machine learning about a year behind.
Messier work is moving too. The Remote Labor Index, from Scale AI and the Center for AI Safety, uses real freelance projects with a median human completion time of 11.5 hours. It counts a project as automated when the AI's deliverable is judged at least as good as the human standard, meaning work a reasonable client would accept. The best agent at launch managed 2.5% of projects. Today's top model, Anthropic's Fable 5.2, reaches 41.3%, about double the next-best models, which matches Christiano's account of the top score doubling in the week before his talk.
Public evidence on AI speeding up AI research comes mainly from METR's assessment of Anthropic. METR found Anthropic's models likely improving on well-specified tasks at about twice the rate of the prior period. It attributed most of that gain to AI, with low confidence, and found no convincing bottlenecks for the next year. Anthropic redacted most of the evidence METR considered most informative, and METR cautions that its findings concern Anthropic and are no timeline for the field. The step from that report to broadly superhuman systems within six months to two years is Christiano's own inference, which he holds with open uncertainty and hopes is wrong.
Warning signs before the means to act
The third lecture, on governance, supplied the argument that gives the series its shape. We may see clear signs of danger before we have any way to respond, and Christiano broke that gap into parts.
Detection is unreliable. Models often know when they're being tested and change their behavior. For drugs, aircraft and financial models, passing a test tells you something about real-world use, and AI breaks that assumption.
The evidence is private. Most of what would tell us how fast things are moving sits inside the labs. Christiano wants company disclosures backed by third-party review and, where the science is unsettled, by open public debate.
Verification barely exists. He listed five things anyone relying on frontier AI needs to know.
- What the system is supposed to do
- Whether it actually behaves that way
- Whether monitoring would catch it if it didn't
- Whether the safeguards hold against attackers
- How capable it is now, and how capable it will be in three to six months
Answering them takes transparency, a shared safety standard, independent verification and enforcement, and most of that machinery is still being built.
Stopping is costly and may come too late. The scenario he considers likely arrives when progress looks dangerous but the models or algorithms may already have leaked. At that point slowing down feels like handing an adversary the lead. Shutting down data centers also does less as capable models get cheap enough to run on ordinary hardware.
Markets and liability won't cover the tail. Someone else's AI can hurt you whether or not you buy it, and the worst-case harms dwarf any company's ability to pay.
His answer was to buy time now, until the means to slow down later are in place. He was clear that he doesn't mean labs sitting idle. Pacing, in his version, means less spending on bigger models and on algorithms that could be stolen, with the money going to safety, security and governance and to putting today's AI to work hardening infrastructure. He pointed to a letter from 1,400 frontier-lab employees, including the chief scientists of most major labs, asking the US government to support an international effort to pace frontier development.
The sponsors offered a useful contrast. Manulife, TELUS and the Global Risk Institute were calm and procedural, describing materiality reviews, in-house testing and guardrails proven before deployment, and Manulife reported $300 million in enterprise value from AI last year. They and the researchers can both be right, since they describe different layers, but the enterprise layer depends on a frontier layer it can't inspect.
Hinton, changing his mind in public
Watching Hinton across the three evenings is a large part of why the series stayed with me. He arrived as the person who usually does the alarming, warning before the second lecture that it would be the darkest of the three. By the third night, in a fireside conversation with moderator Sinéad Bovell, he was conceding that Christiano knows more than he does about what's happening inside the big companies.
His own ideas center on design more than control. He doubts anyone can control something smarter than us, so he wants to build AI that doesn't want to get rid of us. That approach starts with training data curated so models learn good behavior before they see bad behavior. He described an experiment in which training a model on wrong math answers left its math intact and instead taught it that lying was acceptable. Because how smart an AI is and how "nice" it is are roughly independent, he argued, countries could share safety research without sharing the tricks that make models smarter.
Asked what is preventing that cooperation, he was blunt: "a corrupt senile felon who's under the influence of the greedy owners of big tech companies." He added that he doesn't consider China's government wonderful, calling it authoritarian, but believes it currently understands these risks more rationally than the US does. Last summer he met a Politburo member who, in his telling, grasped the case for international collaboration on keeping superintelligence from taking over.
For everyone else, his advice was simple. Pressure politicians to take these risks seriously.
Who can call a halt
My work spans startups, investors, enterprises and students, and most of it centers on capability, meaning what a system can do, what it's worth and what people need to learn to use it well. These lectures left me with one question I now want to ask in all of those settings. What evidence would make us stop a deployment, who would see it, and who has the authority to act?
For founders building agents, the Hugging Face incident makes the evidence question concrete. Any check used to grade an agent is something the agent may learn to game, so the records that would justify a halt belong somewhere the agent can't reach.
For enterprises, Bovell proposed a Sarbanes-Oxley-style signature on model releases and on procurement, with a named person accountable. Manulife answered on the final night by describing its own pre-deployment evaluation. Any company deploying AI can ask today who signs off, and on what evidence. The harder question comes after launch. Someone needs the authority to suspend a system already in production. The triggers for that decision should be agreed in advance, whether a monitoring alert, an incident, or evidence that the system behaves differently when it thinks no one is checking.
Investors face the same question in diligence. Liability and insurance can't absorb tail risk, so the controls matter as much as the coverage. Christiano also noted that rivals catch up through distillation and the spread of ideas, which means a capability lead decays faster than it looks.
For students and educators, the gap sits in the curriculum. Bovell asked Hinton to imagine a quarter of computer science courses devoted to safety and alignment, and he agreed the risk would drop.
If the people building, funding, deploying and learning AI can each answer that question for their own corner, the field moves closer to having brakes that work. As one panelist put it, seatbelts and brakes are what let cars go fast.
Recordings of all three nights are at hintonlectures.com.
