Carnegie Mellon Professor Zico Kolter Leads Critical AI safety Oversight at OpenAI
A Carnegie Mellon University professor is now at the forefront of ensuring artificial intelligence development prioritizes safety, holding the power to halt releases from OpenAI, the creator of ChatGPT, if deemed unsafe. Zico Kolter, head of machine learning at Carnegie Mellon, chairs a four-person panel with the authority to delay or halt the deployment of new AI systems, addressing concerns ranging from potential misuse in weapons development to detrimental impacts on mental health.
“Very much we’re not just talking about existential concerns here,” Kolter said in an interview with The Associated Press.”We’re talking about the entire swath of safety and security issues and critical topics that come up when we start talking about these very widely used AI systems.”
The panel includes former U.S. Army General Paul Nakasone, previously commander of the U.S. Cyber Command, and Altman stepped down from the panel last year to enhance its independence.while Kolter declined to comment on specific instances of the panel exercising its authority, citing confidentiality, the potential for intervention is clear.
Looking ahead, Kolter identified a range of emerging concerns, from cybersecurity vulnerabilities – “Could an agent that encounters some malicious text on the internet accidentally exfiltrate data?” – to the security of AI model weights. He also highlighted novel risks specific to advanced AI, including the potential for malicious actors to leverage AI in areas like bioweapon design or cyberattacks. “But there’s also topics that are either emerging or really specific to this new class of AI model that have no real analogues in traditional security,” he explained.
The impact of AI on individuals is also a key focus. OpenAI has already faced a wrongful-death lawsuit stemming from the suicide of a teenager following interactions with ChatGPT, underscoring the potential for negative psychological effects.
Kolter’s journey in the field began in the early 2000s,when machine learning was a relatively obscure area of study.”When I started working in machine learning, this was an esoteric, niche area,” he recalled. “We called it machine learning because no one wanted to use the term AI because AI was this old-time field that had overpromised and underdelivered.” He acknowledged that even experts underestimated the rapid pace of AI advancement. “I think very few people, even people working in machine learning deeply, really anticipated the current state we are in, the explosion of capabilities, the explosion of risks that are emerging right now.”
AI safety advocates are cautiously optimistic about the restructuring and Kolter’s role. Nathan Calvin, general counsel at the AI policy nonprofit encode, stated, “I think he has the sort of background that makes sense for this role. He seems like a good choice to be running this.” However, Calvin cautioned that the commitments must be backed by concrete action. “Some of these commitments could be a really big deal if the board members take them seriously,” he said. “They also could just be the words on paper and pretty divorced from anything that actually happens. I think we don’t know which one of those we’re in yet.”
© 2025 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed without permission.
Citation: Who is Zico Kolter? A professor leads OpenAI safety panel with power to halt unsafe AI releases (2025, November 2) retrieved 2 November 2025 from https://techxplore.com/news/2025-11-zico-kolter-professor-openai-safety.html. This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part might potentially be reproduced without the written permission. The content is provided for data purposes only.
